Case Study: Fair Isaac Steps Up to a Tape SAN


"At Fair Isaac, we had an opportunity rather than a problem," said Simon Wiltshire, open systems director at Fair Isaac Corporation. Wiltshire had just completed a speech ("Backup to the Future; Implementing a Tape SAN") at Storage Networking World in Phoenix in which he described his team's successful implementation of a tape SAN for backups at Fair Isaac.

Founded in 1956, Fair Isaac Corporation provides predictive modeling, decision analysis, intelligence management, decision management systems, and consulting services to power more than 25 billion mission-critical customer decisions a year. Describing why Fair Isaac needed the tape SAN, Wiltshire referred to Fair Isaac's business needs.

Fair Isaac has been growing quickly, merging last year with HNC Software Inc., adding new clients, and storing more data. "Not only were we growing, but we also found our backup environment was becoming more and more overloaded," explained Wiltshire. "We needed improved reliability and throughput, and also the ability to scale up in capacity."

"Since 1998, we've been migrating significant processing from a mainframe environment to open systems, and we needed to provide greater capability for the open systems backup infrastructure. This has been really successful," asserted Wiltshire regarding his implementation of a tape SAN. But before he described the technology involved (see sidebar), Wiltshire revealed the guiding principles he and his team used:

  • Position the environment for continued growth to ensure a scalable solution
  • Build on existing staff capabilities and ensure operational simplicity
  • Use leading-edge technology but ensure redundancy and reliability
  • Partner for implementation and gain operational knowledge

"This Stuff Is Complex"

Reminding his audience "this stuff is complex," Wiltshire described how Fair Isaac hired consultants from CNT to partner with his Fair Isaac IT staff for the project. CNT is a storage networking solutions provider based in Minneapolis, not far from where Wiltshire was planning to install his tape SAN. "Fibre and tape are still a new enough combination that it's hard to find people with the expertise to plan, install, and run a tape SAN," commented Wiltshire.

However, CNT had an experienced local team that took Wiltshire to visit another CNT customer. The client resembled Fair Isaac in size and in complexity of its transactions and data storage needs. So it was no surprise that the solution CNT had helped them implement resembled the solution Wiltshire had in mind for Fair Isaac. Wiltshire was pleased with CNT's proficiency in this new field -- a key benefit that meant CNT would not be learning about the technology at Fair Isaac's expense. An additional benefit resulted from being able to leverage existing staff capabilities. "I knew my team needed to learn about tape SANs, and it's much easier to learn by working with a knowledgeable teammate," said Wiltshire.

With guiding principles and an experienced CNT team in place, Wiltshire and his tape SAN team established the process they would use. The process tied the project's technical requirements back to Fair Isaac's business requirements:

  • Revisit all backup and archive requirements
  • Categorize by business need with the goal to simplify and consolidate
  • Even out the incremental and full backup schedule by clarifying business unit assumptions and validating service level agreements
  • Schedule and automate to centralize control of all jobs

Page 2: A Viable Process for Businesses of All Sizes

A Viable Process for Businesses of All Sizes

Because it ties all decisions back to business needs, this process would work for a business of any size that is planning to install a new backup system. Its analytical approach removes emotion from decisions, providing results that work immediately and that will scale for future business requirements. "Most of this process is common sense," commented Wiltshire. "But I found it so helpful when the CNT team suggested these measures that I want to point them out to others who might consider a job like this. The suggestion to even out the incremental and full backups to do the same amount of processing daily has especially made a difference in operations."

Wiltshire and his team divided the large and complex implementation plan into two phases and then took about four months to complete them.

During Phase 1, the team confirmed their hardware selection, designed and implemented the hardware infrastructure, prepared the migration plans, and migrated the eight largest database servers using VERITAS NetBackup DataCenter with VERITAS NetBackup Shared Storage Option. They moved about 35 TB of data.

During Phase 2, the team cleaned up the remaining details. They migrated the remaining 220 servers, mostly smaller Windows servers; completed training and handover; and stabilized and tuned the system.

The result of this process was a very successful project. Completed on time and on budget, the project made judicious use of its $1.2 million budget. It increased the reliability of daily backups from 60 percent to 98 percent, and it raised the backup capacity to 25 TB per week, with a capacity of greater than 1 TB per hour. The time needed to complete the biggest database backup decreased from approximately 80 hours to 12 hours. Equally important, the backup tape SAN is well-positioned for scalability.

Lest you think these Fair Isaac tape SAN project team members are superhumans who far outstrip your ability to perform, Wiltshire reports some valuable lessons learned:

  • Have the right expertise available, especially if you're using new technology. For Wiltshire and Fair Isaac, the experts at CNT offered experience and a team approach.
  • Have your team involved in the entire project. Buy-in brings the benefits of clear thinking, problem solving, and dedication.
  • Set clear expectations and guiding principles. Avoid scope creep.
  • Verify assumptions and business requirements. Separate fact from emotion.
  • Plan for the future. Make sure your new installation is scalable.

Sidebar: Technology

» See All Articles by Columnist Marty Foltyn of BitSprings Systems

Sidebar: Technology

Before the tape SAN implementation, the overloaded backup environment at Fair Isaac included:

  • 20 DLT 8000 tape drives
  • More than 3 TB per day full and incremental backups
  • Hardware and tape failures, poor configuration
  • Extensive manual intervention
  • The need for improved reliability, improved throughput, and the ability to scale

To meet these needs, the project team organized a tape SAN architecture based on the following (see figure below):

  • StorageTek PowderHorn 9310 Tape Library, with a 6000 tape capacity, controlled by StorageTek ACSLS software
  • Eight StorageTek T9940A Tape Drives, with native Fibre Channel connectivity and 80 GB capacity (Since the project's completion, Fair Isaac has added four new StorageTek T9940A drives)
  • VERITAS NetBackup DataCenter 3.4, with VERITAS NetBackup Shared Storage Option (SSO)
  • Brocade SilkWorm 2800 Fabric Switches, with 16 ports each
  • Sun Fire V880 Servers
  • Sun Fire 280R Servers, clustered with VERITAS Cluster Server 2.0
  • Largest database servers are attached to the SAN through 1 Gb fibre linking directly to the Brocade Fibre Channel switch fabric. Each server has an HBA.
  • Remaining servers use common backup servers over Gb Ethernet or Ethernet 10/100, backed up through two common media servers
  • NAS storage is backed up through centralized mounts to the media servers (see figure below)

» See All Articles by Columnist Marty Foltyn of BitSprings Systems


Want the latest storage insights?