As last week's Big Blackout of '03 begins to fade into history, businesses are starting to report the first crop of resulting problems with storage and other computer systems. IBM Global Services (IGS) and other consultancies began hearing from customers from last Thursday through early this week, with problems blamed on the outage ranging from lost data to damaged hard drives and "fried" computer monitors. Meanwhile, systems at larger companies like Computer Associates (CA) appear to have emerged from the blackout unscathed.
At IBM's business recovery center in Sterling Forest, NY, the focus was on fast data recovery. "There were some customers who 'declared disaster,'" acknowledges Pat Corcoran, chief of global marketing and business development for IGS's Business and Recovery Services Unit.
Financial firms of various sizes — "all sizes except for the very largest" — arrived at Sterling Forest on Thursday and Friday, often with backup tapes in hand, according to Corcoran. "They needed to recover their businesses, and they couldn't get into their own facilities." The cafeteria at IBM's Sterling Forest plant, ordinarily closed after lunch, stayed open for dinner to feed hungry victims of the power outage.
Sometimes on their own, and sometimes with IBM's help, customers mounted backup tapes and recreated their business environments, relying on data from their last backups. "Some people do backup several times a day, but others do backup only at midnight, for example," notes Corcoran. Other IGS customers use data vaulting.
The Sterling Forest site itself had lost electricity Thursday afternoon, but IBM kept operations running with a diesel-powered generator.
IBM: 'Declared disasters' might have been higher before 9/11
Corcoran speculates, though, that the number of declared disasters from a massive power failure likely would have been much higher a few years ago. Many organizations have taken the lessons of 9/11 to heart, he contends. "Instead of waiting every three years, businesses are updating their business recovery plans a lot more often."
Corcoran also detects a rise in the number of customers with colocated mirrored sites. "These are the people, though, who are the most proactive. Their on-site facilities tend to be fairly sophisticated, too."
'Databases don't like to be shut down'
"I've heard that some companies might have experienced data loss or corruption because of the electrical blackout," says Zachary Slavin, president of The Slavin Group, a business systems consultancy headquartered in New York City. While Slavin didn't mention specific company names, he did say that none of his own customers had been impacted.
Most of The Slavin Group's clients are mid-sized financial firms. Some customers are hosting mirrored sites at the consultancy's data center on Long Island.
"Databases don't like to be shut down when data is being moved. You need to keep the computers online until they can be shut down properly. 'Smart UPS' is best practice. You want the UPS to be able to signal the computer to start shutting down," Slavin advises.
When dealing with critical data, it often makes sense to combine UPS with both electrical generators and mirrored backup sites, according to Slavin. A battery-operated UPS typically runs for an hour or less, while most generators, on the other hand, can operate for up to two days, even without refueling.
Page 2: Double the number of calls for support
IE 7
Firefox 2.0
