Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
It wasn't that long ago that solid state drives (SSDs) were DRAM-based and cost a fortune. Then the proliferation of mobile devices requiring shock and vibration profiles that exceeded hard drives created a huge market for flash. Prices dropped, and with more money available for R&D, the density increased dramatically.
It's been a period of tremendous evolution for flash, but I believe dark clouds are forming on the horizon. Let's start with a little history to help me make my point. I first heard more than 20 years ago that tape was dead, but it took data deduplication to make disk cheap enough to hurt tape sales. So it wasn't disk drives alone that were able to impact tape sales; it was disk drives combined with new technology.
Now we're hearing from some that flash drives are going to replace hard disk drives, and that the cost difference, though great now, will continue to decline. Vendors are putting out charts showing the cost of flash drives and hard disk drives converging, which looks to me like some of the charts I saw for tape and disk more than 20 years ago. I don't think it's going to happen any time soon, and the reasons have to do with lithography limits and disk drive density.
The Limits of Flash Lithography
The main driver of density for flash technology is the size of the lithography used to make chips. I found some good data on Wikipedia, which I confirmed. Here is the change in lithography since 1971, and projected to 2022:
Much of the increase in flash density is related to the reduction in lithography. You can see that from 2002 to 2010, we had almost a two-thirds reduction in lithography size, allowing more things to be fit into the same volumetric space. But it will take another 12 years before we get the same volumetric improvement again, or a 50 percent increase in time. This is why if you chart flash density growth over the last five years, the pace of improvement has slowed.
There are significant challenges to making things ever smaller, which I am not going to address here, but here are a couple of links for further reading: Producing Integrated Circuits With X-ray Lithography and Wikipedia on Electron Beam Lithography. The bottom line is that there are limits to how small things can get with current technology. Flash densities are going to have data density growth problems, just as other storage technologies have had over the last 30 years. This should surprise no one.
And the lithography problem for flash doesn't end there. Jeff Layton, Enterprise Technologist for HPC at Dell, notes that as lithography gets smaller, NAND has more and more troubles — the voltages don't decrease, so the probability of causing an accidental data corruption of a neighboring NAND goes up. "So at some point, you just can't reduce the size and hope to not have data corruption," notes Layton.
Will Disk Drives Grow in Density?
I have been amazed at the number of charts I see that show flash densities growing and disk drive densities stagnating. We all know that drive density growth rates have slowed, but we have still gone from 1TB drives to 2TB drives to reports of 3TB and 4TB drives as soon as this year. We have seen enterprise 2.5-inch drives go from 300GB to 600GB. Yes, flash densities will continue to grow too, but so will disk densities.
Growth rates in storage tend to have periods of fast growth followed by slow growth, then the introduction of new technologies spurs fast growth again. I believe that this cycle, which happens for many technologies, is already happening to flash. The fast growth period from 2007 to 2009 was a trend line that was used to predict an amazing future. Take two or three data points, plot the slope and you have predicted a truly amazing future. It may work for pundits and vendors, but more often than not it does not work in the real world, at least for very long.
Disk drives are going to get denser. Just as perpendicular recording was developed in the early part of the last decade and a growth spurt followed, some new technology such as heat-assisted recording will come along and do the same thing again. The need for more and more data storage at a low cost is not going away, and the demand has not been lost on anyone. Low-cost storage is important for many applications that need to store lots and lots of data, and that's a point that's firmly in favor of spinning disk. If the need for cheap storage were not a perennial demand, then tape would have been dead as predicted more than 20 years ago. Tape is still far cheaper over the long term than disk, and disk is still far cheaper than flash. That won't change anytime soon.