Eight Tips for Implementing a DR Program

Unlike Dorothy in The Wizard of Oz, IT doesn’t have to worry about “lions and tigers and bears, oh my!” Tornados, however, are a shared problem, not to mention hurricanes, earthquakes, blackouts and blizzards. When disaster strikes, it may be tempting to close your eyes and repeat “there’s no place like home,” but unless you have a pair of ruby slippers, the following are better tips to get you safely back to Kansas.

#1 – Distance Matters

Select a disaster recovery location that is far enough away that it won’t be affected by whatever brings your own systems offline.

Florida Hospital, a member of the Adventist Health System, is the nation’s largest privately-owned hospital with 17,600 employees and 2,230 physicians working at 22 campuses. The hospital has its own disaster recovery (DR) site just a few miles from its primary data center in Orlando, but since its primary concern is hurricanes, it also selected a managed SunGard DR site that is 1000 miles up the coast in a location that won’t likely be hit by the same storms.

SunGard has a team of personnel that are knowledgeable in Florida Hospital’s systems and procedures and act in coordination with the hospital’s staff. The cross-disciplinary team of specialists in Windows, mainframe, SAN and networking work together in one room so that, for example, they can quickly determine whether a given problem lies with the server, the network or the storage.

“By being in that same room, we get significantly quicker resolution,” says Bob Goodman, Florida Hospital’s disaster recovery coordinator. “Things that used to take a couple hours to resolve now only take ten minutes.”

Keep in mind, however, the latency caused by the distance between the DR and primary locations. You should know how current each of the backed up systems is and make sure that processes won’t be timing out if they are being run from the remote location.

#2 Keep your DR Plan Current

Hardware, software and configurations can change on a daily basis. Make sure the DR plan reflects those changes.

“Recovery teams are experiencing difficulty in synchronizing their recovery environment to their production environment due to the sheer number of product environment changes,” says John Paul Blaho, SunGard Availability Services’ director of product marketing.

#3 Test the Recovery

Just as backups need verification, so do DR systems, and both must be done on an ongoing basis, not just when initially set up.

Ken Seitz, director of product strategy for Peak 10, Inc., says that most companies look at DR preparation as one-time events and fail to follow through on regular testing. “At the time of a disaster there are a number of problems with the failover,” he says. Sometimes this can be worked through with time, and sometimes they find that the data they wanted just isn’t there.”

#4 Outsource When Appropriate

One of the first points to consider when designing a DR strategy is who is going to do the work: in-house, a service provider, or a mix of the two.

“If an organization lacks the internal skillsets or resources that would be a good scenario to look at outsource DR or a managed service provider,” says Greg Schulz, founder of the Server and Storage IO Group and author of the book Cloud and Virtual Data Storage Networking. “On the other hand, for environments with the skillsets and resources, there can be cost advantages for some organizations.”

#5 Consider the Personnel, Not Just the Tech

When outsourcing disaster recovery, or any IT services, it isn’t enough to select the best technology. Just as important is who is doing the work. Just as you carefully screen those you hire in-house, so should you screen the non-employees you are counting on to keep your business running.

“We believe that most of these [DR] providers approach their offerings through technology alone and not the right combination of great technology, skilled and experienced people, and tried and tested processes,” says Seitz. “When the moment of disaster declaration comes you want someone piloting the system and the process that has gone through it many, many times and not figuring it out as he/she goes.”

#6 Don’t Let a Lack of Time Put off Your Planning

When you’re dealing with “disasters,” it is easy to put off planning against some future disaster that may possibly occur. If you don’t have time to fully research and develop a plan, hire an expert to do it for you.

Although many customers are acutely aware of their risks, most have historically been unable to enhance their disaster recovery plan due to a lack of time to focus on the initiative or a lack of funding to build a recovery environment, says Jaclyn Mispagel, sr. consultant, Windstream Hosted Solutions.”

#7 Archive Any Unnecessary Data

Paying to have a second set of unnecessary data ready for immediate recovery is a waste of the DR budget. Set up and enforce storage tiering policies, and only store the data that will be needed at the DR site. Do the same with servers: don’t pay to have a second set of zombie servers.

#8 Take Care of Your Own Personnel

Executives are always saying it is the employees that make the company great. It is more than just a PR line. Don’t rely solely on personnel from the area impacted by the disaster to take care of the data: let them focus on making sure their children are safe as their first priority. If you do send some of the IT staff out of town to a DR location, send their families with them.

“We realize it is all about people and processes, and take a very human approach,” says Goodman. “All our employees that deploy can take their families, with the hospital covering the hotel and airfare. It causes a lot of stress and a feeling of helplessness to leave the family behind.”