Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
MOUNTAIN VIEW, Calif. Symantec demonstrated a new Object File Server technology it said its researchers have developed to help deal with the management and storage issues IT is facing in dealing with the massive growth of unstructured data. The amount of data large enterprises and IT service providers are required to store is growing by 50 percent a year, the company said.
"We would see this being used in large enterprises for things like video surveillance, activity logs, engineering documents, all the unstructured data that's growing like crazy, not your typical Oracle database," said Joe Pasqua, vice president of research for Symantec Labs, in a media briefing here at Symantec's (NASDAQ: SYMC) headquarters.
OFS provides a new architecture that leverages commodity components standard storage racks, Ethernet switches and data servers. In addition to data servers, a metadata server in the rack acts as a traffic cop managing access to up to thousands of clients.
"We scale by adding more data servers. Once you've filled up a rack, add a new one and that looks just the same, there's no special master, they all have the same architecture," explained Pasqua. "You could have multiple servers totaling petabytes of data."
Pasqua said the file system is architected for resiliency. "We know there is going to be failure in any system. OFS stores files as objects, there are redundant copies we balance the load across different servers. If a disk fails, OFS rebuilds the objects automatically."
He said the rebuild is a key point of differentiation compared to traditional file systems that have to go through a number of checks following a failure that could delay access by hours if not days in very large systems.
Eliminating Data Storage Bottlenecks
"The cool thing is we only rebuild data, not empty space," he said. "The fact is that a lot of low-level systems don't know what empty space is. A disk block is a disk block to them, so they move it. We do know so there's no single disk bottleneck."
On the security front, OFS requires that every client be authenticated and authorized to access file systems. The system requires that every single read and write be digitally signed to protect information against unauthorized access. Access is managed by the metadata server.
For administrators, Symantec claims an OFS system will be much simpler than conventional systems because there is no need to configure disks or RAID groups or rebalance loads. A single administrator of an OFS system could automate file system mounting for thousands of clients.
Symantec has no timetable for when OFS might be coming to market. The company said potential markets include providing the technology to hardware partners and storage service providers.
Symantec said the OFS prototype advances the scalable file server concept it introduced with the debut of FileStore last fall. FileStore is designed to let organizations build scalable, high-performance file storage services for their enterprises, including private and public clouds.
OFS is designed to address the growing need to store more information in the cloud with scalability, availability and cost requirements in line with those demands. Symantec said OFS can support clusters containing up to eight petabytes of physical storage capacity organized in one or more file systems containing up to one billion files each.
Follow Enterprise Storage Forum on Twitter