Enterprise Storage Forum content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
DigitalGlobe may not be God, but the company has got the whole world in its hands, or rather its ImageLibrary.
Since January 2002, DigitalGlobe’s QuickBird satellite has been collecting hundreds of thousands of images of our planet, which are stored in DigitalGlobe’s ImageLibrary and sent to Google — for use on Google Earth and Google Maps — and other international customers via media and FTP transmission.
To be able to process the data-intensive raw images collected by QuickBird, turn them into usable imagery and safely and securely store both the raw and finished files required a special kind of storage and data management solution.
As DigitalGlobe’s IT director, Luc Trudel, explains, “Storage plays a critical role in DigitalGlobe’s production line and asset management. As all of our imagery products are digital in nature, we need high-performance disk storage for processing raw imagery into finished products as well as cost-efficient tapes for long-term storage of raw and processed imagery.”
In addition to wanting a system that had the ability to store potentially hundreds of terabytes of media rich data, DigitalGlobe needed a solution that could work within its “highly heterogeneous” computing infrastructure, which includes Linux (SUSE and Red Hat), Solaris, IRIX and Windows operating systems.
After looking at several storage and data management solutions, DigitalGlobe went with the StorNext file system from Advanced Digital Information Corporation (ADIC) — technology that Quantum will soon acquire in a merger announced late Tuesday.
“It was the only solution we found that provided the scalability, performance, features and operating system coverage that we required,” says Trudel.
“Most file systems won’t go over 2TB, but ADIC’s goes over 100TB,” adds John-David Childs, DigitalGlobe’s lead network manager. That’s a good thing, since DigitalGlobe plans on launching a second satellite in 2007 that will triple or quadruple the company’s storage needs.
A Data Management and Storage Solution in One
ADIC, a leader in the open systems market and the largest supplier of automated tape systems, specializes in providing companies like DigitalGlobe with data management and storage solutions that can work with both very large data files and disparate operating systems.
“What we want to do is to take these disparate systems and tie them together, so they can all share a common set of data, and we do that with a shared file system,” explains Nathan Moffitt, ADIC’s StorNext product marketing manager. “So instead of having to pass data from system to system over the network, we allow everyone to tie into one common pool of storage and access that data very quickly, typically over a Fibre Channel SAN, but it could also be done by iSCSI, so that they can get higher levels of efficiency.”
“With DigitalGlobe,” says Moffitt, “we were taking all these different processes within their workflow — ingest, distribution, processing — and we allowed them to access a single repository of data so that they could finish their projects in three to four times a shorter a period of time. That’s really one of our biggest advantages, allowing customers to consolidate their data into a common pool, so they don’t have lots of pools of direct attached storage (DAS), which are inefficient and seldom used very properly.”
Indeed, with StorNext’s high-speed data sharing, users are able to access files from multiple hosts up to four times faster. Additionally, StorNext’s multi-tier archiving gives users continued access to data while reducing storage costs. And ADIC’s iMover technology transparently moves data between storage tiers, including external appliances. ADIC says the software is typically installed in less than half a day and does not require any end user training.
“The real beauty of the product is that the interface is the same as what every application understands,” says ADIC software architecture manager Steve Johnson. “The application just opens and closes files directly to that new drive letter, which can vary from computer to computer.”
Who Can Benefit from StorNext?
While DigitalGlobe has seen image processing times improve fourfold since deploying StorNext, the software isn’t for everyone. ADIC’s two primary markets are companies such as DigitalGlobe that have large shared media files and want to optimize workflow, and companies looking for a flexible multi-tier storage system.
“When we look at high-performance workflows, customers are concerned about speed,” says Moffitt. “We actually have an install running right now that’s going over 15GB per second, and we can push data really really fast into [a centralized disk] repository. As time goes on, it makes sense to not have 200TB worth of spinning Fibre Channel disk, and we’ve turned to many customers and said, ‘Well, let’s reduce the amount of high-end storage you have and let’s move that data that’s not being used as often off onto a cheaper, more cost-effective storage medium.'”
That’s where ADIC’s tiered storage solution comes in.
“We have an HSM add-on component that allows us to transparently migrate data from the centralized disk repository out to a tertiary storage medium,” says Moffitt. “That can be cheaper disk. It can be a NAS appliance. It can be a tape library. It can even be vaulted off out of a library if the customer so desires.”
That flexibility helps clients control costs while still giving them access to their data.
“There are many aspects of imagery storage: processing performance, online availability across multiple storage tiers, long-term archiving, scalability and manageability,” explains DigitalGlobe’s Trudel.
When choosing a storage solution, he says, “be clear on which aspect of imagery management you’re addressing and then find the solution that best addresses that problem through extensive pre-sales testing. Clearly it’s desirable to try to find a vendor that covers all of these aspects, but don’t compromise on too many aspects just for the sake of working with a single vendor. This is where a good VAR/integrator is worth its weight in gold.”
DigitalGlobe works primarily with Advanced Systems Group.
For more storage features, visit Enterprise Storage Forum Special Reports
