Five years have passed since Hurricane Katrina, one of the most deadly and costly hurricanes on record, devastated the Gulf Coast. While the Gulf Coast is still recovering from the damage caused by the 2005 hurricane season, the National Oceanic and Atmospheric Administration’s (NOAA) Climate Prediction Center is warning of a 2010 hurricane season eerily similar to 2005. Citing all-time high sea surface temperatures in key areas of the Atlantic Ocean, as well as the El Niño cycle, NOAA predicts that the 2010 season will most likely produce 14 to 23 named storms, eight to 14 hurricanes and three to seven major hurricanes (Category 3 or stronger).
Faced with this alarming forecast and with Hurricane Earl gaining strength off of the Eastern Seaboard, many state and local agencies are investigating what they can do to keep operations up and running during a major disruption such as a hurricane. Although most agencies are aware that a disaster preparedness plan is important, many key components are often either overlooked or not fully tested. Developing a solid and proven disaster preparedness plan is crucial to ensuring minimal interruptions to critical operations during weather emergencies or other disruptions. The following business continuity checklist can help state and local agencies take the first steps to avoid costly downtime, reduce inconvenience to constituents, and prevent disruption of critical services provided by the organization to the public:
Assess your current plan and prioritize essential services. Even large agencies that boast large budgets in boom times are hard pressed to cost-justify a disaster preparedness plan that includes every agency process. In lean times, and for small agencies with limited resources, developing a comprehensive plan can be even more challenging. Agencies should conduct a business impact assessment that prioritizes critical processes for the entire organization.Agencies should tier services in order of the time by which they must resume to minimize the impact to operations (recovery time objective). For example, processes that need to continue immediately to prevent serious mission impact, such as essential citizen services, could receive an “A” rating. Processes that can be continued within 24 hours could receive a “B” rating, followed by “C” functions that can be restored within 72 hours, and so on.
Take steps to protect data and enable access. Aside from people, information is the most critical asset for almost every organization. Just as agencies should set a recovery time objective for mission critical services, so should they rank the impact of lost data. The greater the impact, the more often organizations should back up their data. Agencies might even consider upgrading backup equipment to a faster version to reduce the time required to complete a backup cycle, and they might even back up continuously for mission critical data. Multiple copies of data should be stored off site, at a remote location, far from the primary data center.Agencies sometimes focus so much on protecting and backing up data that they often neglect to take steps to ensure their employees can remotely access that data if the office is closed due to a disaster. Remote-access software provides employees with access to networked server or desktop information outside of the office.
Review power and cooling systems. Agencies should add uninterrupted power supply (UPS) to keep the most essential applications running. A UPS battery backup unit serves as a bridge to keep IT systems running until main power is restored, a secondary power source is deployed, or until the IT system can seamlessly be shut down. In addition, agencies should assess the projected growth of the organization and ensure that enough UPS devices are available to accommodate that growth – ensuring that all systems are protected in emergencies today and in the future.Cooling systems should also be supported by backup generators, as data centers can heat up quickly if equipment operates on backup power without adequate cooling. Temperature spikes can reduce the lifespan of network equipment by half and also cause unplanned system interruptions when operations are most critical.Installing a power backup system does not eliminate the requirement to regularly inspect and maintain the power infrastructure. Agencies should periodically ensure that automatic transfer switches are configured so that there is a little lag time to disrupt UPS power to computer systems. They should also conduct regular battery inspections and replacement. Like flashlight and smoke detector batteries, UPS systems need to be inspected before they are needed.
Identify and appoint a cross-functional preparedness team. Agencies should create a team from various departments (e.g., applications development, servers and systems administration, network operations) to identify and prioritize critical processes, design and test the disaster recovery plan, and select an outside service provider. A recovery team, which will participate in recovery activities after any declared disaster, should also be developed. While the recovery team can be similar to the cross-functional preparedness team, the recovery team should not be identical, even within a small agency. Additional members of the recovery team should include IT executives, outside service provider representatives and even community members.
Document, test, and update. The cross-functional preparedness team should develop a disaster preparedness plan that includes logistical details, including directions to backup sites and a list of people who have spending authority for emergency needs. The plan should clearly identify the role of each individual on the cross-functional preparedness and recovery teams and be tested in an environment that simulates an actual emergency.During testing, the written plan should be followed as closely as possible, and leaders should ensure all designated individuals carry out the action items assigned to them. Leaders will be able to spot problem areas that cannot be identified by just reading the plan and update the plan accordingly before a real disaster occurs. Because organizational structure and operations change frequently, it is important to review the plan frequently to ensure any gaps are bridged.
Consider telecommunications alternatives. Essential to any agency’s disaster preparedness plan is a contingency plan for telecommunications. Alternative communications vehicles, including wireless phones and satellite phones, should be considered. Agencies should become familiar with their telephone service provider’s emergency power capabilities and may want to investigate auxiliary power sources such as an uninterruptible power supply or battery backup, coupled with a surge protector. Agencies should also consider call forwarding and establishing an 800 number to improve accessibility during a disaster.
Form tight relationships with vendors. Hardware, software, network, and service vendors can help expedite recovery, as these contacts often can ensure priority replacement of telecommunications equipment, personal computers, servers, and network hardware. Strong vendor relationships are especially important for small agencies, which may lack the resources that larger organizations can tap in an emergency.
Because change is constant within most agencies, and because organizations are increasingly dependent on information systems, it also pays to revisit each step on a regular basis. Products and services designed to help during emergencies also change, as do their methods of delivery. A business continuity plan must keep pace with these changes for it to be useful in the event of a disruptive emergency, and tests of the preparedness plan must be conducted regularly to ensure readiness.
David Hutchins is director of state and local government sales at CDW Government (CDW-G), a leading source of IT solutions to governments and educators. Hutchins is responsible for leading the team of sales executives and account managers focused on meeting the unique needs of CDW-G state and local customers.