It's been observed throughout human history that you generally have to get kicked in the teeth before you see value in the art of self defense. It certainly seems to be true in the case of disaster recovery, where vendors experience far more interest in their DR wares immediately following events such as 9/11, Hurricane Katrina, or more recently, floods and tornadoes.
"We see spikes of interest from time to time when an event occurs, and that does drive more communication with our customers," said Brian Regan of IBM's information protection services unit.
Recent alerts from the National Oceanic and Atmospheric Administration (NOAA) to expect a normal or near normal hurricane season certainly didn't result in businesses suddenly rushing to dust off those neglected purchase orders for that state-of-the-art business continuity (BC) setup they began to scope out in the aftermath of Katrina. A recent study by Aberdeen Group found that 34 percent of companies have yet to implement any kind of DR solution. Of the remaining 66 percent, 25 percent don't perform regular disaster plan tests. And when you get down into the SME sector, Jeffrey Hill, an analyst with Aberdeen Group, reported that nearly half of those in the 100 to 1,000 employee category don't have a BC/DR plan in place.
Cost or Complacency?
The situation isn't solely based on complacency. Many of the companies involved want to be able to recover rapidly in the face of an event. But they just can't afford it. Say you wanted to set up a large-scale backup/recovery architecture such as that used by Salesforce.com. Salesforce has to have the type of backup and DR platform that doesn't lose a single transaction no matter what happens. That takes a total of nine copies of the data, each with different recovery points for failure. The company's complex array of tape libraries, disk arrays, servers and databases cost around $20 million. Continuous data protection (CDP) is achieved via regular shadow images of production data shared between mirrored SANs on separate coasts. Oracle databases are also continuously protected. Other scenarios are deployed ranging from 4 hours to recover all the way to 48 hours for less critical systems.
Of course, not everyone needs that level of protection. Accordingly, multiple degrees of DR have evolved, ranging from the "Remain live or die" category through more prioritized approaches on down to economy class.
Let's look at what may be considered basic some kind of regular backups occurring along with a plan to recover systems and data within a reasonable time. It's important to differentiate these two.
"It's hard to imagine that a company wouldn't have a data backup system, but just backing up data alone doesn't constitute a disaster recovery strategy," said Hill.
Yet as the Aberdeen numbers prove, many don't make it beyond backup to the point of instituting some kind of DR plan. And on the other extreme, you have those who are so fed up with the hassles of backup technology that they have implemented DR without any kind of underlying backup setup.
The Department of Consumer Affairs of the City of New York, for instance, uses StorageX from Brocade specifically for DR. Its main office in Manhattan stores data on an EMC Celerra NS500 NAS box. Its other major site in Queens has another NAS filer. Users at Queens actually write to the EMC box and then everything at Manhattan is replicated to Queens using Microsoft DFS. Brocade StorageX aggregates the file data into one logical file system. If Manhattan goes down, users are directed to Queens automatically.
So confident was the IT department that it conducted the ultimate test unplugging the EMC box in the middle of a work day. Even more surprisingly, the department does not even use a regular backup system. It relies on this DR set up to safeguard all its data.
"When we tested it, everything worked beautifully," said Matthew Miller, the department's LAN administrator. "The system failed over to the secondary location and users didn't notice a thing."
At the other end of the spectrum comes Pella Corporation, a manufacturer of windows and doors based in Iowa. It runs its business on Oracle E-Business Suite applications and Oracle Database 10g using a centralized architecture. It also deployed a remote data center for recovery known as the bunker.
"Prior to the bunker, we had a lot of equipment for component failure and documented plans for a site disaster, although it would have been a lengthy process had we lost our data center," said Jim Thomas, Pella's director of IT operations.
HP supplied its StorageWorks XP24000 arrays, one at each of the two data centers. Both systems are interconnected using Fibre Channel to replicate continuously between the two locations. The bunker is hardened to protect against tornadoes, and it has the ability to run all vital systems if the primary data center is destroyed.
That effort took several years and multiple millions to establish. But you don't necessarily have to spend in that order of magnitude to protect your systems. Lipscomb & Pitts Insurance LLC, based in Memphis, went the appliance route. This type of solution fits best for SMBs or organizations that are shorthanded, or for larger organizations that don't want to bring in further DR/storage expertise.
Lipscomb & Pitts purchased the appliance from STORServer of Colorado Springs. The company's vice president of finance, Mike Yates, said the move wasn't prompted by the experience of a particular event.
"You do not know when a disaster will happen, so you have to make contingency arrangements," he said. "If you lose your data, your business shuts down and you cannot serve your clients."
The 100-employee company uses a combination of EVault InfoStage and EVault InfoStage DualVault from Seagate, all operated by masterIT, an IT managed services firm from Tennessee. It looks after Lipscomb & Pitts' backup/recovery and IT operations. Over 250 GB is backed up each night.
Yates says bunkers, massive automated tape libraries and expensive replication gear are fine for large enterprises companies, but are out of the question for many SMEs.
"For a fraction of that cost, we're just as prepared and didn't have to pay an arm and a leg," he said. "It's a comfort knowing the data is going out every night to a different region in a way that is largely transparent to our staff."
No Time to Waste
There are many different strategies and technologies that can be employed when it comes to DR, all at different price points. The choices made are often based upon the cost of achieving various recovery point objectives (RPO) and recovery time objectives (RTO).
"If you have an unlimited budget and resources, an RTO and RPO of zero or close to it would be great," said Greg Schulz, senior analyst and founder of StorageIO Group. "But in reality land for many environments, implementing the right tier of protection to applicable threat risks is a means to maximize budgets and spending."
Schulz suggested that small and mid-sized businesses look to protect essential and time sensitive material with the lowest RTO/RPO they can afford while looking at how to leverage different RPOs and RTOs for other data.
Whatever DR scheme is required, though, it should be deployed sooner rather than later.
"SMBs cannot wait until a disaster of company-ending proportions occurs before allocating the resources to do this right," said Aberdeen's Hill.