Disaster Recovery: IT Pros Handle Hurricane Sandy - Page 3
Dimension Data: Helping Employees
Having employees prepped for a disaster is about a lot more than spinning up new servers in the event of a failure. IT services company Dimension Data’s office in Framingham, MA was affected by the hurricane, though not nearly as bad as many others. The office lost power, but they rolled over to a generator and were fine.
However, employees were working long hours taking customer-service calls. For some employees, this meant that Dimension Data had to rent hotel rooms across the street from the office, so they cloud get some quick rest. For others, it meant equipping their home offices, so those locations could stay online.
“Our corporate strategy is to give employees as much flexibility as possible,” said Darren Augi, VP of IT for Dimension Data. “Our workforce can really work from anywhere. Our home office employees are just as ready to go in a disaster as the in-house ones are.”
The reason the remote employees are just as ready is because Dimension Data made this a priority, by ensuring they had redundant means of communications, and even going so far as to issue portable generators suitable for powering sensitive electronic devices.
One thing Sandy drove home, though, is that communications infrastructure in the U.S. is falling behind other developed countries. “In a disaster scenario, 3G is just not enough for employees,” Augi said.
The other thing Augi stressed is that organizations must remember that employees are making major sacrifices to work during this time. Many of them have been affected, and you have to respect that.
Key points: Plan to help employees get through the hard work as they deal with the disaster by, for instance, providing nearby hotel rooms; give remote workers the tools they need to stay online; remember that it’s not just your organization and customers struggling through the disaster, but your employees as well.
iCIMS: Protecting Internal and External Resources
SaaS provider iCIMS managed to keep its talent acquisition solutions online despite the storm, with none of its 1,300 clients losing service. With fully redundant data centers in North America, UK and China, its decentralized IT model served it well.
However, the big lesson learned was that it needed to provide its internal business applications with the same level of protection as its external applications.
With corporate offices in Matawan, NJ, just a couple of miles from the Atlantic shore, damage to an electrical substation meant that the offices were without power for a week.
“Even though we had a week-long disruption, we managed to keep our sales operations in place enough that we continued to add customers,” said Naveed Chaudhry, a network admin at iCIMS. “However, we suffered on our internal applications.”
iCIMS discovered that conducting regular full backups of corporate data and transporting tapes offsite every week may not be a failsafe strategy in the face of Mother Nature. “We thought this was foolproof but it didn’t work, as we didn’t have power,” Chaudhry said.
He noted that having a generator internally wouldn’t work for a company of the size of iCIMS. A generator would provide power in a pinch, but it would require upkeep and wouldn’t really be economically feasible. In any case, conditions were so bad that even those attempting to commute to the office faced impassible roads.
Therefore, he coped by helping personnel to operate remotely. The company already had a number of remote staffers and they largely carried the load. But even this presented hurdles. The company had half a TB of data on the cloud with Zetta, an enterprise-grade cloud storage provider. After talking with Zetta, Chaudhry arranged for the backup service to overnight a backup disk, as that would be faster than trying to download the data over the Internet.
Initially, the plan was to set up a new server and restore the entirety of the corporate data at another location where power was available. But tight time constraints required the company to prioritize what got restored.
“When we got the disk, we decided to only restore those portions our remote users required to operate,” he said. Moving forward, the plan is to emulate its external application strategy.
“Our best option is to have our core corporate services outside of the office – move all critical applications at headquarters to a data center and make them fully redundant like our SaaS apps,” Chaudhry said. “The lesson we learned is to decentralize our internal operations just like our external ones.”
Key points: Internal operations must be covered by DR plans as well as external data and apps – decentralization is valuable; in a pinch, your cloud backup provider can send you a disk for expedited restoration; a backup generator may or may not provide necessary ROI.