As last week's Big Blackout of '03 begins to fade into history, businesses are starting to report the first crop of resulting problems with storage and other computer systems. IBM Global Services (IGS) and other consultancies began hearing from customers from last Thursday through early this week, with problems blamed on the outage ranging from lost data to damaged hard drives and "fried" computer monitors. Meanwhile, systems at larger companies like Computer Associates (CA) appear to have emerged from the blackout unscathed.
At IBM's business recovery center in Sterling Forest, NY, the focus was on fast data recovery. "There were some customers who 'declared disaster,'" acknowledges Pat Corcoran, chief of global marketing and business development for IGS's Business and Recovery Services Unit.
Financial firms of various sizes — "all sizes except for the very largest" — arrived at Sterling Forest on Thursday and Friday, often with backup tapes in hand, according to Corcoran. "They needed to recover their businesses, and they couldn't get into their own facilities." The cafeteria at IBM's Sterling Forest plant, ordinarily closed after lunch, stayed open for dinner to feed hungry victims of the power outage.
Sometimes on their own, and sometimes with IBM's help, customers mounted backup tapes and recreated their business environments, relying on data from their last backups. "Some people do backup several times a day, but others do backup only at midnight, for example," notes Corcoran. Other IGS customers use data vaulting.http://o1.qnsr.com/log/p.gif?;n=203;c=204655439;s=10655;x=7936;f=201806121855330;u=j;z=TIMESTAMP;a=20400368;e=i
The Sterling Forest site itself had lost electricity Thursday afternoon, but IBM kept operations running with a diesel-powered generator.
IBM: 'Declared disasters' might have been higher before 9/11
Corcoran speculates, though, that the number of declared disasters from a massive power failure likely would have been much higher a few years ago. Many organizations have taken the lessons of 9/11 to heart, he contends. "Instead of waiting every three years, businesses are updating their business recovery plans a lot more often."
Corcoran also detects a rise in the number of customers with colocated mirrored sites. "These are the people, though, who are the most proactive. Their on-site facilities tend to be fairly sophisticated, too."
'Databases don't like to be shut down'
"I've heard that some companies might have experienced data loss or corruption because of the electrical blackout," says Zachary Slavin, president of The Slavin Group, a business systems consultancy headquartered in New York City. While Slavin didn't mention specific company names, he did say that none of his own customers had been impacted.
Most of The Slavin Group's clients are mid-sized financial firms. Some customers are hosting mirrored sites at the consultancy's data center on Long Island.
"Databases don't like to be shut down when data is being moved. You need to keep the computers online until they can be shut down properly. 'Smart UPS' is best practice. You want the UPS to be able to signal the computer to start shutting down," Slavin advises.
When dealing with critical data, it often makes sense to combine UPS with both electrical generators and mirrored backup sites, according to Slavin. A battery-operated UPS typically runs for an hour or less, while most generators, on the other hand, can operate for up to two days, even without refueling.
Double the number of calls for support
Meanwhile, ActionFront Data Recovery Labs received about 260 phone calls on Monday, twice the typical daily average of 130 calls. ActionFront's offices are located in Buffalo, Atlanta, Chicago, Santa Clara, and Toronto. Almost all the calls represented "new business," reports Nick Majors, company president.
"A lot of businesses were still closed on Friday. Even if a business was open, people were still trying to reconfigure systems by themselves last week. By Monday, though, our phones started ringing all day along."
Most callers reported damaged hard disk drives, but others pointed to compromised RAID systems or glitches in "advanced operating systems."
Majors suspects, though, that some reported computer troubles are not really blackout-related at all. "It's possible that some people are blaming problems on the power failure, just to cover up for their own failures," he theorizes.
CitySoft, Inc. also received twice the usual number of calls at the start of this week. David Rosenthal, president of the New York City-based IT consultancy firm, says he expected to get calls about hard drive problems stemming from abrupt shutdowns. A number of callers, though, got in touch with CitySoft about "fried" computer peripherals.
"At one company, all monitors from one particular manufacturer got fried. At another, all printers from a different manufacturer got fried. I can't really blame the vendors, though. I think there must have been some sort of a big power surge about a microsecond before blackout. This type of thing would be outside the vendors' usual design parameters."
Computer Associates has a mirrored data center — but didn't need it
Big systems vendor Computer Associates has a pre-established backup site in the Midwest that mirrors its main data center at company headquarters in Islandia, NY.
"We are headquartered on Long Island. If we'd gone 50 miles west (of Islandia), the mirrored site would have been located in New York City. So the company made a decision that if we had to go farther west anyway, we might as well put the site in the Midwest," says Walt Thomas, CA's CIO.
The Midwest site is colocated at another CA facility. Above and beyond their usual job duties, staff at the Midwest facility are trained to take over emergency backup data center operations.
As things turned out, CA didn't need to resort to the mirrored data center at all last week. The lights came on in Islandia at 4 AM on Friday morning.
Until then, CA temporarily depended upon on-site electrical generators. "If we'd run out of fuel before the power came back on, we could have then refueled the generator — although whether we could have gotten the supplies from the fuel companies is another question," says Thomas. "On the other hand, it's more costly to implement a mirrored data center than to use a generator. These are the kinds of tradeoffs you have to weigh."
An Ounce of Prevention...
All in all, disaster prevention measures and recovery systems appear to have held up remarkably well in their most widespread test in recent memory. With damage to critical corporate data apparently nominal for most companies, the tried and true adage of "an ounce of prevention is worth a pound of cure" appears to have proven its point once again.