HP Hood's IT Department Milks Outsourced Backups
HP Hood's IT department has closed its server backup window forever, sliced yearly backup expenses in half, and stopped singing the backup blues. How? Outsourcing!
With yearly sales of $500 million, HP Hood, a 150-year company based in Chelsea, Massachusetts provides frozen deserts, citrus, non-dairy, and specialty food products throughout New England and the mid-Atlantic states.
About 550 employees use the 15 Windows NT servers, a mix of IBM NetFinity 550s and 500s, and Compaq ProLiant 550s, distributed across the six offices in New England, and two offices in New York. A WAN connects all these offices.
Located in five of the offices, six systems administrators for almost a year and a half used to fire off the daily incremental backups and weekly full backups for each server. However, backing up more than 100 gigabytes of storage onto the eight HP SureStor DAT devices, one located in each office, became a daily lesson in spilt milk.
William J. Moulton, Jr., HP Hood's IT network manager, says, "We'd start at 8 p.m. to do a full backup of the corporate server's 35 Gbytes. When my systems administrators came in the next morning, the backup would still be running. We'd have to kill it so employees could use the server without any latency issues."
Backing up Windows Exchange Servers using Computer Associates' ARCserve, the backup software, proved unreliable. Moulton says, "A lot of times, backups would fail and there wouldn't be anyone around at 3 a.m. to switch the tape. At 8 a.m., we'd have to re-start the backup just as employees are getting on the network."
Moulton put manual fail-safe procedures in place. Before the six systems administrators could handle other tasks in the morning, they had to verify that the backup run on the servers assigned to them. If it didn't, they had to investigate the cause - the tape, the backup software, or the way the backup was scheduled. All findings went to Moulton each day. He says, "It took each one an hour and 15 minutes each day to verify their servers. The task could take longer if they had trouble logging into a remote server or a lot of employees were using the server."
Saying Goodbye to Backups Forever
In April 1999, Moulton declared a moratorium on the Windows NT backups. He decided to pay $45,000 a year and outsource the entire task to amerivault corp., a Waltham, Massachusetts-based startup offering a business-to-business online backup and restore service to companies in New England! The service enabled Moulton to freeze his yearly backup expenses by half. His yearly expenses included $20,000 for tapes, $8,000 for the software licenses, and $81,000 in wasted manpower (9 hours per day x $35 an hour). Two budgeted expenses included $28,000 to replace the DAT devices, and $8,000 to upgrade ARCserve for Windows 2000.
Moulton says, "The systems administrators can schedule the backups any time and from any location, and amerivault does all the work. They now begin their mornings by making sure the servers and the network are up, not worrying about backups."
Unlike tape backup which copies the entire file each time, amerivault shrinks HP Hood's backup window by transparently taking only the changes since the last backup for that server. When HP Hood first came online, amerivault did a full backup of HP Hood's files and from this created a mapping strategy in an index file. Each time a backup occurs, amerivault looks for files that have changes, and then locates those changed blocks, compresses and encrypts them, and sends them over the wire.
To use ARCserve, the systems administrators completed a 40-hour training session. With amerivault, they installed the amerivault agent software on their workstation or laptop and on each server. The software provides the step-by-step procedures for scheduling even a series of staggered backups.
Moulton says, "They can walk away or watch as the backup occurs. They can see which files have changes, and how much data was compressed. If the backup fails the first time, they'll get e-mail message immediately."
HP Hood's backups take anywhere from five minutes to 20 minutes the most, rather than several hours. Employees can use the servers as they are being backed up. Moulton says a server can take a slight performance hit during the backup.
To transmit the backup to amerivault's DS3 line, HP Hood uses its full T1 Internet connection, which goes to a port outside of the company's firewall. amerivault stores the files live on SCSI-disks on two RAID subsystems - a LSI Logic MetaStor RAID and a Dell PowerVault. Encryption methods range from 128 Blowfish to the federal government's Tempest.
Previously, HP Hood's systems administrators would have to search ARCserve's backup log to locate the tape, then load the tape in the DAT device, and then go back and find the file and restore it. Moulton says, "They prayed the tape worked.)
To restore files with amerivault, systems administrators select the files, directories, or entire drives they want to restore from, and then they answer a couple of questions about where to place the data. Finally, their server then connects to amerivault and forwards the files to the designated location.
Moulton says, "Amerivault provides the peace of mind that my backup and restores will never fail. If the service cost more a month, it would still be worth it."
Elizabeth M. Ferrarini - is a free-lance writer from Boston, Massachusetts.