Disaster response: the good, the bad and the ugly

By Sascha Meinrath, Government Technologies

Contrary to popular perception, the problem of disaster recovery is often not the lack of resources, but lack of coordination.

One key component to successful emergency response is a dynamic, direct and robust communications network -- a structure the United States had been missing. Key decision-makers turned a deaf ear to the problem until Hurricane Katrina made such an ostrich-stance untenable, and the United States had to learn the lesson the hard way. Yet a year later, improvements have been incredibly modest. During the next major disaster, experts say we should expect more of the same -- a lack of coherent, rapidly deployable, interoperable communications networks for first responders and the communities they serve.

In many ways, the state of U.S. disaster response is not too different from what we see in far less developed areas of the globe. Following the magnitude 7.6 earthquake that struck Pakistan, India and Afghanistan on Oct. 8, 2005, many problems faced by first responders were eerily similar to those experienced in Katrina's wake. According to one Indian IT expert familiar with the situation, "The machinery of government had difficulty getting and sending even a handful of satellite phones for use in the devastated areas. I don't know if any of them have fully ready-to-move transportable (airliftable) satellite video uplinks, which would certainly be very useful. Similarly equipment for receiving remote-sensing imagery in real time and GPS/location equipment [was lacking]."

Jeff Allen, a consulting engineer currently working in Liberia with Médecins Sans Frontières, was a key member of Radio Response and the Community Wireless Emergency Response Initiative following Hurricane Katrina. Both groups developed and deployed critical telecommunications and network infrastructure in the hurricane's aftermath.

In terms of U.S. scenarios for emergency communications and disaster response, Katrina provided a sobering example of what works, what doesn't work, and the lessons we could learn from the ensuing massive communications meltdown. Allen's on-the-ground experiences helping to coordinate telecommunications disaster recovery were presented to the FCC on March 6, 2006.

What Worked, What Didn't

Generally speaking, hands-on investment in disaster preparedness is both sorely needed and relatively lacking. Designing networks to be deployed in advance is one of the most valuable lessons disaster recovery workers learned. Caching equipment and training recovery teams are also critical to these efforts. Yet more than a year after the largest natural disaster the U.S. has ever faced, little has been done to improve communities' preparedness.

During disaster recovery, one of the most important elements is the organization of human beings. Thus, current initiatives to create separate infrastructures for "official" responders and the rest of the community are met with skepticism by those who have worked on the ground. "I have heard some vendors talking about municipal networks with VLANs [virtual local area networks] for public access and VLANs for public safety people," Allen said. They tend to treat the public access as an add-on, or as a luxury that can be turned off when bandwidth gets tight. That's lunacy. Giving people the tools to work together and solve their own problems is way more powerful than giving 20 police cars full motion video over a wireless Ethernet system. Humans need low bandwidth and existing collaboration systems hosted out in the network to organize to help themselves."

Instead of improving communications, new emergency response systems often expand the gulf between responders and the communities they are supposed to help by creating additional technological barriers to shared use.

And while the technology stories that emerged from Katrina often focused on the glamour of certain services (e.g., video streaming, VoIP), tried and true applications like instant messaging (IM), private chat rooms, and Web access actually worked best. "VoIP is OK for networks that are fully controlled, and whose topology and capacity are well planned," Allen said, "but it's unusable on the kind of agile, fluid and low-bandwidth network you find in a disaster area."

Unfortunately more municipal networks are being built using both hierarchical and centralized infrastructures -- all but guaranteeing they will be more prone to failure during a disaster. "A closed system that depends on a proprietary configuration server would be dead in the water when the configuration server lost power (a common occurrence in a disaster area)," according to a Radio Response report.

Likewise, newer communications technologies, some of which were first deployed during the post-Katrina disaster recovery, often turned out to be nonfunctional. Though many press releases from major corporations heralded the successes of WiMAX and mesh, first-experiences were different. "In Mississippi I did not see any indication that mesh technology works," Allen said." I don't know why not. The arguments for it are compelling and the technology has had enough time to be stable, it seems. It might just be that deploying enough working nodes and keeping them working is the problem."

Disaster recovery nationwide is often facilitated through incident command systems (ICSs) that respond to everything from a single house fire to a Katrina-sized event. While ICS training is accessible online and through the Federal Emergency Management Agency (FEMA), few communities have made this training available to emergency responders and interested residents. One important element of ICS is the emergency support facilities (ESF), which consist of numerous divisions that each deal with a different facet of disaster recovery. Though the goal of these divisions is to coordinate responses, often in-fighting and personal politics led to perception of them as a barrier to effective on-the-ground recovery. For the Community Wireless Emergency Response Initiative, this meant interfacing with ESF-2 (Communications), ESF-15 (Volunteer Coordination) and ESF-5 (Facilities), and getting approval from each for different elements of their work.

During emergency response, one should expect a lack of advanced technological know-how from most emergency responders. As documented for the FCC in the Radio Response report, "it seems that most people who handle radios for emergency operations do not understand electronics, physics or RF [radio frequency] propagation ... they consider any nongovernment use of RF equipment a threat to their turf. People coming from this point of view are rarely swayed by facts or by regulations." Time and again, officials dismiss innovative solutions to emergency communications because they don't understand the technologies.

Allen laid out four simple recommendations to the FCC to help facilitate successful telecommunications setup during disaster recovery, proposals that a municipality of almost any size could easily establish:
* Preplan the network architecture, including "what if" scenarios for how to modify the architecture in response to a situation.
* Maintain a cache of preconfigured and known-good hardware.
* Take part in real-life drills using the real cache hardware.
* Have at least one experienced staff member on call to provide leadership and continuity during an actual disaster response.

The Radio Response team recommends easily deployable hardware. "The devices have to act like simple appliances. Configuration should be via Web user interfaces. If we are to have a dynamic routing system, it must work in the home networking context." In other words, plug-and-play, off-the-shelf hardware is often preferable to far more specialized (and often more expensive) solutions. Unfortunately numerous Community Wireless Emergency Response team members found themselves spending too much time troubleshooting donated equipment, and worrying about relatively unimportant logistical details and potential legal issues.

Network engineers sporadically found that the available hardware equipment was unstable, broken or in need of software upgrades. In addition, maintaining an up-to-date, "known-good" equipment cache is critical because much of the equipment on the market is problematic out-of-the-box. "The quality of the engineering of the software -- and to a lesser extent, hardware -- is very low in these types of devices," Allen said. "Software bugs are very common, and unless you are using a particular 'blessed' version of the firmware, behavior is far from predictable."

Furthermore, during the Katrina disaster recovery, obtaining legal Windows licenses for refurbished computers installed in refugee centers proved impossible, and responders often resorted to using illegal copies of the software. As a result, one of the recommendations made to the FCC was to utilize nonproprietary systems whenever possible. In the case of computer operating systems, responders recommended Linux LiveCDs, which let a computer boot directly from the CD itself, and allow users to quickly burn more operating system disks as needed. "Fetching, burning and running a LiveCD is practical in a disaster context," wrote Allen. "Debugging is not. We wasted a significant amount of time with configuration errors. It is easy to make them in the context we were working in, and it was exceptionally difficult to find them and fix them."

In disaster recovery, the "official playbook" doesn't always conform to on-the-ground realities. "Both FEMA and Red Cross depended to a huge extent on telephone service working," Allen said in his report. "Their behavior in this regard was strange, as it seemed to disregard the reality that close to 100 percent of the victims from Hancock County were without reliable personal telephone service." Once emergency responders restored Internet connectivity, often well ahead of phone service, storm survivors found even more disturbing practices from the major players. For example, FEMA's Web site required a relatively recent version of Internet Explorer (IE) to complete forms to receive federal aid. Users who ran Linux, Macintosh or other IE-incompatible operating systems couldn't apply for FEMA assistance online.

Better Today Than Tomorrow

After Katrina, Community Wireless Emergency Response Initiative team members like Mac Dearman, Will Hawkins, Joel Johnson and Paul Smith focused on coordinating and setting up telecommunications infrastructure as quickly as possible in incredibly chaotic environments.

Enthusiasm, adaptability and a MacGyver-esque ethos were often the most important elements to the successful completion of the day's tasks. This often means telecommunications recovery efforts need to revisit previous work sites to make network upgrades. Allen put it this way, "You have to learn that when you are operating day by day on what could charitably be termed a 'good plan,' you must schedule time later for rework, to incorporate the unknowns the 'good plan' glossed over. This is true in all network design, I think, but it is a bigger deal when the cycle time is so short; a network built last week might be ready for significant rework this week. This is a common problem in the emergency management context." During disaster recovery, one often doesn't have the luxury to wait for orders or directions from higher-ups. In addition, the authorities don't always know how to get things done better than the local community does. "If you expect to get direction -- or even accurate intelligence -- from the authorities, you'll be disappointed," said Allen.

Internet infrastructure is surprisingly low on the priority list during disaster recovery. As the Radio Response report stated, "The priorities are transport (without which you can't move resources to solve any of the other problems), then communications, then survival commodities like water and (later) food. Communications is a very high priority, but the needs are met with a small set of linked VHF repeaters and stand-alone satellite connections, not with an Internet distribution network." Though reprioritization of this critical resource will eventually happen as government agencies realize the importance of Internet connectivity, for now, most communities end up cut off from all Internet-reliant services for long periods of time whenever disaster strikes.

Public access isn't the only reason to restore Internet connectivity as fast as possible. If post-Katrina network activity statistics are any indicator, the networks set up by the Community Wireless Emergency Response Initiative saw an equal amount of usage from disaster relief workers themselves. Responders often used the network to communicate with their home bases, since cell phone coverage was often spotty and landline communication completely nonexistent. Workers also used the network to depict the disaster's impact on daily life. "Individuals used the Internet connection to explain what they were experiencing to friends back home," Allen said. "They sent out e-mail to worried parents and posted to blogs. Sharing their experiences like this helped attract more volunteers and resources to get the job done."

The most common use of computers connected to these networks was Web-based e-mail (Hotmail, Yahoo, Gmail, etc.). However, users also downloaded IM clients to communicate in real time with friends and family. Additional examples of network usage included:
* coordinating with supporters back home to bring more supplies and volunteers;
* remotely accessing workplace computers;
* paying bills;
* searching for new jobs;
* complaining about current conditions;
* posting news and commentary; and
* checking prices on eBay for selling salvaged collectibles to raise cash for house repairs.

Solutions and Lessons Learned

Many in the emergency management community recommend public investment in open standards and nonproprietary, interoperable technologies. While VoIP and other services are exciting new tools, the most critical applications are the latently strong ones like e-mail, Web access and IM. "Volunteer community teams" are also as fundamental to disaster recovery.
Data collection and management are also important, and critical information should be easily accessible to responders:
* All geographical information -- maps, GPS readings, driving directions -- is needed by responders. Often, fundamental assumptions need to be shifted. For example, relying on street maps is useless when signs are blown down, and referring to landmarks is sometimes the best option.
* Contact information for team members, current and prospective key customers/survivors, government and aid organization liaisons, and contractors is often overlooked or difficult to access. Best contact mediums also shift often during disaster response.
* Network information -- IP allocation plans, currently assigned network numbers, network diagrams, passwords, administrator contacts -- is also rarely accessible or systematically tracked. Ensuring that important communications network information is current and available will help save time -- and avoid headaches. Redundant and robust networks such as satellite uplinks, wireless, landline connections require more information than many traditional single-medium system architectures.
* Equipment information -- notes on configurations, hardware and cabling, vendor manuals and inventory -- is vital. One of my favorite stories is about "Brent" who brought boxes of one-gallon heavy-duty Ziploc bags to save storage space and reduce the time spent searching for the correct wall plug or cable by letting responders package equipment -- e.g., router, Ethernet cable, power brick and manuals -- into transparent, waterproof, easily opened units.
* Documentation -- timelines, photos, press, video, notes -- is often downplayed during disaster recovery. Yet learning from previous initiatives is vital to continued improvement of recovery efforts.

Most disaster recovery responders said priority is often placed on "outside experts" and "professional services," but the successful mobilization of local community assets is also critical to disaster recovery. For example, many community teams are perfectly situated to help build and interconnect telecommunications networks, and often have the social capital necessary to bring diverse constituencies to the table.

Practice, preparation and preplanning were hailed as fundamental to facilitating smoother operations during the chaos of disaster response. "Help communities take care of themselves," Allen said. "It's respectful of people's need to be involved in their own recovery. It gives you huge leverage with small investments, and it lets the best aspects of American culture -- like teamwork, ingenuity and giving -- overcome the worst situations."

Sascha Meinrath coordinated the Community Wireless Emergency Response Initiative and is also the founder and executive director of CUWiN.net. He serves on the board of directors for CTCnet, a U.S.-based network of more than 1,000 organizations committed to improve the educational, economic, cultural and political life of their communities through technology. In 2006, he founded EthosWireless.com, a wireless consultancy focused on social justice.

article originally published at http://www.govtech.net/digitalcommunities/story.php?id=102914.

The media's job is to interest the public in the public interest. -John Dewey