Enterprise Storage Forum content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Faced with the daunting task of providing citizens, banks, insurance companies and other authorized users access to more than 20 million discrete images residing in its Filenet Document Management System, the County of San Bernardino Auditor-Controller/Recorder’s Office turned to a clustered storage solution from Isilon Systems.
The Isilon IQ storage system and Isilon SyncIQ Replication Software provide the high level of throughput, speed, reliability and scalability the local government required, according to Patrick Honny, department of information systems administrator for the County of San Bernardino Auditor-Controller/Recorder’s Office.
“Today, our Isilon storage provides us with a level of reliability that’s a quantum leap in improvement from our previous system,” says Honny.
The Auditor-Controller/Recorder’s Office houses all official land, birth, marriage and death records filed within the county electronically in a Filenet Document Management System.
Just five years ago, millions of images were stored on a CD-based jukebox. “We had an in-house system that talked to the jukebox,” says Honny. The system eventually succumbed to the vast number of transactions moving in and out of the system on a daily basis.
At the time, according to Honny, the government department was processing a couple of thousand of transactions per day. The speed was unreliable, often resulting in a several minute wait time. “Additionally, as transactions built up in the queue, the system would shut down,” he says.
The county office then brought in network-attached storage (NAS) from Procom Technology, which was acquired by Sun Microsystems in June 2005. Honny reports that the 8 terabyte NAS system worked out well, noting that system speed was adequate and scalability wasn’t an issue. The county also looked at NAS, SAN and server-centric storage at the time.
The downside of the solution, he points out, was that it was prone to single points of failure. “Even though we had redundant controller cards, we had card failures,” he says. The system was also prone to hard drive failures because the NAS couldn’t handle certain file sizes, he notes.
Ultimately, though, it was a controller card failure that resulted in data loss and knocked the system out of commission for over 24 hours that had the county searching for a new solution.
Time for a Change
Frustrated with the Procom NAS, it was by chance that Honny noticed a magazine article on clustered storage sitting open on a desk in his department. “All I knew was that I didn’t want to be in the same position again,” he says, referring to the 24-hour outage.
That’s when the county began to do some homework.
The IT team called in four vendors who offered a clustered storage solution. “At the time, about 18 months ago, the concept of clustered storage was fairly new,” says Honny. Isilon stood out among the competition.
“We talked with the company president about the maturity of its products as well as the viability of the company,” he says.
When all was said and done, Honny’s discussions with Isilon’s executives led him to the conclusion that the vendor had the best funding and best products. “We were told that the clustered storage could handle multiple failures across nodes and still be accessible,” he says.
The next phase of investigation was to undertake an in-house trial.
The county office brought in an Isilon IQ three-node unit, each unit with three terabytes of storage for a total of nine terabytes. Honny targeted a demo unit that would be the same capacity as his initial production unit.
“We loaded it up with over a terabyte of production data and put it through a trial by fire,” he says, testing the Isilon IQ unit with potential worst-case scenarios. For example, a node was unplugged while the system was in operation; a network connection was unplugged; and the backbone channel connection was loosened.
“During this phase, we continued to test the response times and there were no problems,” says Honny. “The time had come to move the Procom NAS out and move the Isilon IQ solution in.”
Moving Forward
San Bernardino County purchased a 9 TB Isilon IQ system about a year ago. Since that time, they’ve grown the system to 15 terabytes by adding two 3 TB mirrored nodes. The clustered storage unit was loaded up with data over a weekend.
“Our Filenet document management system has 20 million discrete images, four to six pages each, for about one hundred million pages of data. The system handles the data in 16 gigabyte chunks,” says Honny.
For high throughput and high concurrency, the county opted to use Isilon IQ supported high-performance InfiniBand for intracluster communications. The second option for a communications backbone is gigabit Ethernet. In its initial purchase, Honny also included an extra mirroring system and the Isilon SyncIQ Replication Software.
According to Honny, Isilon engineers were helpful throughout the demo and the test phase. The vendor’s engineers also worked with county IT staff to help train on the SyncIQ policy-based file replication management tool and Web-based interface.
“But for the most part, the system is plug and play,” he says.
Since installing the system, the county has had only one issue with the Isilon solution, and that has to do with the mirroring capability. “We’re currently working with Isilon on how to better deal with sending file changes over the WAN/LAN,” says Honny, explaining that currently, an entire 16 gigabyte file from the Filenet document management system is sent over the network. “Isilon is helping us figure out how to send over the changes only,” he says.
Just recently, the county began using the Isilon IQ storage for shared files. The department’s users had been using network-based server-centric storage for shared files. The Compaq HP DL380 servers were used as file servers across two servers. Rather than replace the overloaded servers, the IT department opted to move the storage for shared files over to the Isilon system.
“We moved about 300 private drives of about 100 gigabytes each and about 100 shared drives of about 500 gigabytes each,” he explains. Expanding the file servers would have involved major surgery because they’re not scalable. “All we had to do with the NAS is buy another 3 TB node and take 10 minutes to install it,” he says.
The only remaining Isilon project is to move the department’s SQL database to the NAS. Currently, the database is stored on clustered servers that have about a three-year life cycle. “They currently have about a year left. Then we’ll move it over to the NAS,” says Honny.
For more storage features, visit Enterprise Storage Forum Special Reports
