Disaster Recovery: IT Pros Handle Hurricane Sandy - Page 2
BUMI: Submerged Under Seawater
You’d think that if anyone would be ready for a storm like Hurricane Sandy, it would be a disaster recovery firm like BUMI. The BUMI internal DR team held a conference call the day before Sandy hit New York and determined that this storm had the potential to be really, really bad.
“To get through this storm, we realized that we had to become customers of ourselves. We invoked the highest-level DR plan that we offer for our customers,” said Jennifer Walzer, CEO of BUMI. The night before the storm hit, the team decided to move their online presence to data centers in Toronto and Vancouver.
It was a good thing they did. When Sandy blasted New York City, BUMI’s office building was submerged in 35 feet of seawater. They lost power and as of late November still had no access to their office building. But because their servers failed over to a data center in Canada, they were able to remain operational in spite of not having a physical office.
Additionally, BUMI’s servers were at Verizon’s central office, which was completely submerged, with all of its copper wire destroyed. BUMI may have had to move both its office and its central data center.
BUMI relied on VoIP phones, so once communications started to fail, the VoIP system failed over to employee cell phones. “That was the good news,” Walzer said. “The bad news is that we didn’t think to have Cisco phones in everyone’s homes.”
Walzer told BUMI’s operations manager to FedEx Cisco VoIP phones to employees who needed them, but for a few days, all of BUMI’s customer support – and this was a peak period for support, obviously – was handled via cell phones.
“That was a big lesson for us,” Walzer said. “It’s not enough to just be able to work from home, you have to be able to work from home as if you were working from the office.”
BUMI was able to keep all of its customers happy, yet working as hard as they did took its toll on staffers.
“You need to be able to lean on your coworkers, and that’s just not something you plan for,” she added. Since BUMI has no physical office space at the moment, employees felt out of the loop. They needed face time with coworkers, so they started meeting up once a week in the city.
This need for employee communication isn’t something you typically plan for ahead of time, but if you don’t realize it as you’re coping with a disaster, employee performance can suffer.
Key points: You need to plan for being out of your office for weeks or months, not just days; being ready to work from home doesn’t mean you have a “home office”; the social connections among coworkers are important and must be supported.
SGFootwear: Choosing a Backup Site
Shoe manufacturer SGFootwear was another organization that learned how dangerous it is to have your backup site in the same geographical region as your primary one.
SGFootwear’s main site is located in Hackensack, NJ, while its backup site is a mere 12 miles away in one of its warehouses in Kearny, NJ. Sandy brought down all the communications lines to the main site, but otherwise there was no damage to the building. However, the backup site was flooded.
“We originally chose the backup site for easy access,” said Gregg Asch, Director of IT at SGFootwear. For critical backups, the company uses the backup solution from Nimble Storage. “We figured that if anything went wrong, it wouldn’t take long to get down to Kearny, grab the Nimble device, bring it back to our main site, and we’d be back up in no time. Obviously, if we would have studied flood patterns, we would have made a different choice.”
Now, SGFootwear plans to move its backup site to a more geographically distinct location, with the eventual goal of having a mirror site across the country in Los Angeles. The other lesson that Asch hopes his organization learns is the value of mobility. Currently, SGFootwear discourages telecommuting and is wary of BYOD – but this needs to be reconsidered. “This is something I’ll be stressing as we do the post-mortem on this event,” Asch said. “We should be more decentralized and less reliant on any one location.”
One final point Asch stressed was the value of virtualization. Their Nimble environment is a virtualized deployment. “If our main site had been trashed and the backup survived, we would have been able to move the Nimble device, spin up the VMs and get going. You don’t need like equipment in both sites. Virtualization makes the whole process easier,” he said.
Key points: Choose sites to be not just geographically but topologically distinct; telecommuting and BYOD should be part of your DR plans; virtualization makes recovery easier.