With its data storage needs soaring nearly 10-fold in less than five years, Baptist Memorial Health Care (BMHC) knew the time had come to address storage challenges such as backup, disaster recovery and archiving applications. Today, the medical institution not only boasts improved data recovery times, but it also has a cost-effective, fast and reliable storage system in place designed to meet both short and long-term requirements.
Since 2001, the Memphis, Tenn.-based hospital network's data has grown from 1.5 terabytes to 138 terabytes, driven by technologies like medical image archiving, electronic patient records and electronically stored laboratory results.
The amount of information to be stored became huge, according to Hal Weiss, assistant engineer in the server management group at BMHC. "Trying to protect the data was the real challenge," he says, noting that 2004 became the year for implementing a robust data protection system in line with standards such as the Health Insurance Portability and Accountability Act (HIPAA).
Considered one of the largest non-profit hospitals in the country, BMHC is a network of 15 hospitals throughout the mid-south. Boasting $2 billion in revenue, the medical institution employs about 12,000 people and has more than 83,000 admissions annually.
Gaining Control
The new objective for BMHC, according to Weiss, was protecting data and recovering it in a timely manner.
"It was taking us 30 days to back up to physical tape our 9-terabyte Patient Folder System," he explains.
Trying to restore all of the information in the 384 million-file system, containing individual files with 55KB or less each, would take more than 60 days in the event of a disaster.
"Those times were unreasonable," says Weiss.
The IT group had a number of methods that it could deploy for better data protection. However, only one of its three options had any merit, according to Weiss.
One method was based on replicating cabinets on the SAN. "The drawback to this method was the potential for data corruption," says Weiss. If data was corrupted on the cabinet, the mirror would be corrupted, as well, he explains.
The option of using physical tape had its drawbacks, from how to handle, store and move it, to sensitivities to temperature and other problems related to Mylar. Furthermore, says Weiss, "Historically, 20-25 percent of anything backed up using this method can't be retrieved."
Weiss set his sights set on virtual tape library (VTL) technology and continuous data protection (CDP) methodology.
Moving Ahead
In 2004, IT staff brought in both CDP and VTL products to test. The CDP test went on for a month, the VTL test for 21 days. Both tests ran financial systems and one clinical system. The VTL ran on HP UX and Windows.
The results were impressive, according to Weiss. On the VTL, the backup window was reduced from six hours to a little more than three. Failure rate, for both backup and restore, was reduced to zero from 25 percent, on average. Furthermore, with CDP, BMHC was able to recover to any point in time. "We forced a file corruption and were able to recover in 15 minutes. The system reacted extremely well," says Weiss.
On the CDP, the test system had a 60GB Oracle database structure which could be restored to any point in time within 15 minutes.
After the testing, Weiss had only a few issues to address. The only issue regarding the VTL was floor space. "We needed to find a solution that provided a lot of storage in a small footprint," he says. For CDP, his mission was to find a solution that gave continuous recovery capability, was not file-based, and was hardware agnostic.
On the CDP front, Revivio offered the block-level hardware protection that Weiss was looking for.
While more than two dozen vendors provide VTL, all using different media for disks and different sizing, only one vendor enabled the maximum amount of data storage in the smallest footprint. That vendor was Copan Systems.
"We looked at Quantum, EMC and Copan," says Weiss. The Quantum and EMC solutions required about five cabinets, each with a 6.5 square footprint, or about 30 square feet. Copan, by contrast, offered 224TB in a single cabinet in a 10 square footprint. "On top of size issues, there were power and cooling requirements to consider," adds Weiss, noting that the Copan solution required only two power leads and cooling.
Copan offerings quickly rose to the top and became the solution of choice for BMHC in January. The solution consisted of four shelves with 112TB of space, with expansion for four additional shelves that come matched with drives for future expansion. The health care facility paid approximately $300,000 for the system, a cost that is comparable to buying a tape library, according to Weiss.
However, he expects to see a return on his investment in two years based on tape replacement alone. "We're also able to do backup faster and we're not missing doing our backup because we've run out of hours in the day," he says. Savings on one staff person's time is estimated at about 10 work days per year.
According to Weiss, BMHC backup consists of 4.5TB daily, which includes everything on all 384 of the data center servers — e-mail, financials, patient records, and file and print data. Of the 4.5TB, 2TB of the info stored daily is retained for two years. The other 2.5TB is kept for 30 days.
Six hours after the product arrived, Weiss was backing up data. "We were surprised at how easy it was to set up," Weiss says. Management is also easy. "There are tools, but the system basically maintains itself," he says.
In fact, when a Fibre Channel card failed early on, the system generated its own e-mail to Copan, as well as a 6 A.M. call to Weiss. "Before we knew it, Copan came in and swapped out the failed card. Our system performance was never impacted," he says.
For more storage features, visit Enterprise Storage Forum Special Reports
IE 7
Firefox 2.0
