Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Over the last year we've looked at different rates of increases for CPUs and storage, which show improvements in storage are clearly lagging behind those of processing power. IDC, Gartner Group, and all the industry pundits agree that storage needs will grow faster than the increase in CPU power. Even with the recession, storage is still growing as a percentage of the overall IT budget at most companies. It will likely continue to grow, and perhaps at an even greater rate than today given that:
- Most companies are not supposed to delete data given regulations enforced by the U.S. Government, SEC, European Community, and others
- Decision support such as POS, data mining, and other retail tracking tools
- Every drug we take, plane we fly on, car we drive, and numerous other items that are computationally modeled have all the model runs saved both for reason number one above as well as to help with potential lawsuits
- New areas have emerged with huge data volume requirements, including earth imaging, climate modeling, weather forecasting
- U.S. Government applications such as Homeland Security, science and engineering, and other research have similar data storage requirements
The Recent Past
I am sure some of you heard about a company that was going to be the end all, be all of storage. It was called Terastor and was funded by some industry heavyweights to the tune of over half a billion dollars. In 1998 they were going to produce a 10 GB read/write optical disk that could write at 15 MB/sec and would soon follow with 20 GB and 100 GB disks by 2000, if my memory is correct.
As you can imagine, they failed, as did a number of other companies that were trying to develop optical technology, both tape and read/write CDs. DVD is a consumer product making some inroads to the computer storage market, but this is still a consumer product with different specifications than are required for long-term data storage.
Meanwhile, disk density has been steadily increasing every 18-24 months at about two times increase. I remember 4 GB drives in 1995, with 9 GB in 1996, then 18 GB in 1998, and now with Fibre Channel, 36 GB, 72 GB, 143 GB all spinning at 15K RPMs, and 180 GB spinning at 10K RPMs.
When will we see 283 GB drives or 360 GB drives? The trend for density doubling seems to be taking longer. Yes, every few months we hear claims of laboratories increasing disk density, but moving things from the laboratory to the production lines often takes a long time.
Another area that has yet to catch up is software protocols. The SCSI standard was designed around 32-bit processing, hence the "Small" in SCSI. SCSI is limited to 2 terabyte LUNs, which will become a problem with 283 GB drives, as for RAID-5 8+1, that would be 2.2 TB. And with 360 GB drives you are over the limit with a 6+1 RAID-5 configuration. The standard will change, but how soon will the server and storage vendors adopt the new standard, and how soon will you be able to upgrade your server software to support the change?