Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Management, File Systems Lag
Two problems remain, however: management and file system technology are not where they need to be.
We are truly lacking an enterprise management framework. Yes, SNIA and others have APIs and specifications for management, and many RAID and switch vendors have followed them. Vendors that write management software are busy adding devices and features, but we are not even close to end-to-end management, which in my opinion not only includes simple things such as configuration of devices, but also includes error analysis and host and file system control. Some vendors might have a framework that works with their products, but even the error analysis is a hard issue to deal with.
One site I work with has more than 1,000 switch ports in a single SAN. Tracking HBA issues, SCSI retries, cyclic redundancy check (CRC) issues and performance issues is not simple. Maybe I should not be able to buy a commercial off-the-shelf package for a site of this size with multiple file system types, different switches, and host types, but I should be able to get closer than we are, and storage management software and network management frameworks are not getting closer fast enough.
With file system technology, I believe we will see incremental increases in scaling and performance at best. In the first half of next year, a number of companies will come out with hardware and software products that will provide security, storage performance management and remote replication. With the lack of a major increase in network performance and a major price drop, we will still have the same problems that we have today, with too much data to replicate given the data creation rate and the speed of the WAN.
Late 2005 and Beyond
The farther out in time we go, the greater the risk of making a prediction that turns out to be wrong. There are many potholes on the road to getting a product to market, including poor design architecture, ASIC design problems, software testing problems, firmware problems, manufacturing problems, supplier problems, products that are not released within the written specification, and many other possible problems. You may not be aware of it, but all of these problems happened to various vendors this past year to delay products, and they happen almost every year. So predicting which vendors will meet their schedule is difficult, but here is a list of what I see coming in the second half of 2005.
10Gb Fibre Channel RAID devices: QLogic, McData and other vendors already have 10Gb switch-to-switch connections, so support for other 10Gb connections is not going to be a huge technology jump.
Terabyte tapes: Imation has already announced plans for 0.5 TB tapes and a roadmap to 2 TB tapes. If you can make the tapes, then companies are going to be able to make the drives.
Vendor products based on the T10 Object Storage Device standard should become available late in the year or early 2006. Now that the OSD specification has been out for about six months, we should start seeing products in a year or so. This is SOP in the industry.
We should see 600 GB ATA drives. Since HDS has already announced 400 GB SATA drives, 600 GB is not much of a stretch. This trend with larger and larger drives will continue, with SATA always being greater in size than FC/SCSI drives. This could have a long-term impact on the cost of FC/SCSI drives, since SATA drives are taking some of the market share away from FC/SCSI.
I also think we'll see better methods for hardware integration of SAN to WAN to SAN for remote shared file systems.
On the software side, I think we'll see: better management tools but not nirvana; better Linux cluster tools, but no silver bullet; better HBA drivers and file system options for Linux; and better shared file system integration for heterogeneous environments.
For 2006, my crystal ball starts getting a little blurry, but here are some best guesses anyway:
- 1 TB tapes, with either LTO standards, IBM or StorageTek.
- OSD shared file systems and object storage manager (RAID) integration with HSM will become available or announced.
- PCI Express in large servers with more than 24 CPUs. This is a big issue, since the memory backplane will need to be redesigned.
- More usage of RDMA-TCP. This could change the nature of storage devices, since we might live in a TCP-IP world even for storage if this takes off.
- Finally, and most importantly, we will finally get some real security for file systems, and for storage in general. This might include encrypted file systems with real user interfaces, and management that runs at device rates.