Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
It wasn't that long ago that solid state drives (SSDs) were DRAM-based and cost a fortune. Then the proliferation of mobile devices requiring shock and vibration profiles that exceeded hard drives created a huge market for flash. Prices dropped, and with more money available for R&D, the density increased dramatically.
It's been a period of tremendous evolution for flash, but I believe dark clouds are forming on the horizon. Let's start with a little history to help me make my point. I first heard more than 20 years ago that tape was dead, but it took data deduplication to make disk cheap enough to hurt tape sales. So it wasn't disk drives alone that were able to impact tape sales; it was disk drives combined with new technology.
Now we're hearing from some that flash drives are going to replace hard disk drives, and that the cost difference, though great now, will continue to decline. Vendors are putting out charts showing the cost of flash drives and hard disk drives converging, which looks to me like some of the charts I saw for tape and disk more than 20 years ago. I don't think it's going to happen any time soon, and the reasons have to do with lithography limits and disk drive density.
The Limits of Flash Lithography
The main driver of density for flash technology is the size of the lithography used to make chips. I found some good data on Wikipedia, which I confirmed. Here is the change in lithography since 1971, and projected to 2022:https://o1.qnsr.com/log/p.gif?;n=203;c=204650394;s=9477;x=7936;f=201801171506010;u=j;z=TIMESTAMP;a=20392931;e=i
Much of the increase in flash density is related to the reduction in lithography. You can see that from 2002 to 2010, we had almost a two-thirds reduction in lithography size, allowing more things to be fit into the same volumetric space. But it will take another 12 years before we get the same volumetric improvement again, or a 50 percent increase in time. This is why if you chart flash density growth over the last five years, the pace of improvement has slowed.
There are significant challenges to making things ever smaller, which I am not going to address here, but here are a couple of links for further reading: Producing Integrated Circuits With X-ray Lithography and Wikipedia on Electron Beam Lithography. The bottom line is that there are limits to how small things can get with current technology. Flash densities are going to have data density growth problems, just as other storage technologies have had over the last 30 years. This should surprise no one.
And the lithography problem for flash doesn't end there. Jeff Layton, Enterprise Technologist for HPC at Dell, notes that as lithography gets smaller, NAND has more and more troubles — the voltages don't decrease, so the probability of causing an accidental data corruption of a neighboring NAND goes up. "So at some point, you just can't reduce the size and hope to not have data corruption," notes Layton.
Will Disk Drives Grow in Density?
I have been amazed at the number of charts I see that show flash densities growing and disk drive densities stagnating. We all know that drive density growth rates have slowed, but we have still gone from 1TB drives to 2TB drives to reports of 3TB and 4TB drives as soon as this year. We have seen enterprise 2.5-inch drives go from 300GB to 600GB. Yes, flash densities will continue to grow too, but so will disk densities.
Growth rates in storage tend to have periods of fast growth followed by slow growth, then the introduction of new technologies spurs fast growth again. I believe that this cycle, which happens for many technologies, is already happening to flash. The fast growth period from 2007 to 2009 was a trend line that was used to predict an amazing future. Take two or three data points, plot the slope and you have predicted a truly amazing future. It may work for pundits and vendors, but more often than not it does not work in the real world, at least for very long.
Disk drives are going to get denser. Just as perpendicular recording was developed in the early part of the last decade and a growth spurt followed, some new technology such as heat-assisted recording will come along and do the same thing again. The need for more and more data storage at a low cost is not going away, and the demand has not been lost on anyone. Low-cost storage is important for many applications that need to store lots and lots of data, and that's a point that's firmly in favor of spinning disk. If the need for cheap storage were not a perennial demand, then tape would have been dead as predicted more than 20 years ago. Tape is still far cheaper over the long term than disk, and disk is still far cheaper than flash. That won't change anytime soon.
Flash Performance Not Always Worth the Price
Flash performance for small block I/O (less than 32 KB reads) is orders of magnitude faster than disk, but we all know that flash drives are not created equal for writes. Some are orders of magnitude faster than disk and some are maybe 10 to 20 times faster, and for that speed you pay a significant cost. The question is whether the cost justifies the expense.
Let's take a typical home PC application such as Photoshop. We all know that file sizes are growing, as cameras have ever larger sensors. A few years ago the file might have been 2 MB, but today it might be 8 MB or larger. Reading an 8 MB file on a typical SATA hard drive might take — with a seek, latency and read — 0.15 seconds, but on a flash drive maybe 0.1 seconds, depending on the drive. Does that minute difference really matter? What about editing a large video file? The disk drive might be 80 MB/sec, while the low-end flash drive might be 100 MB/sec. Again, not a huge difference. What does matter is that the metadata used for the editing application is going to go way faster on a flash drive. I know many people with more than 500 GB of data at home between music, videos and pictures. The space requirements add up very quickly. People are not going to spend even 50 percent more for small performance increases at home.
Back in the early 1990s, I had a tape backup system for my home PC. Tape for home PCs is a thing of the past, never to return, but I don't see flash hurting disk to the extent that disk has hurt tape sales. Flash technology won't achieve the densities of hard drives at anywhere near the same cost, at least anytime soon. There are just far too many challenges. Flash and hard drives will have to coexist, and if flash vendors don't see that, they will risk missing their target market, as cheap bulk storage is needed for most data just as much as small block random I/O is essential for a few use cases.
My next PC at home, for Photoshop editing, personal finances and long-term storage, will be a Windows PC with a small flash drive with a home NAS box with four 2TB hard drives connected over 10 Gb Ethernet (at least that's my hope). I will use the flash drive for booting and for temporary files for Photoshop editing, and the bulk NAS storage for pictures, videos, music and everything else.
Hybrid storage requirements are here to stay. Flash addresses the problem for small files, small application I/O requests and swapping far better than hard drives ever will. Flash will never have the price/density advantage of hard drives, however. Solid state technology will have a significant impact on hard drive companies, but flash alone will never be able to meet storage density requirements. Welcome to the new tape vs. disk argument, and expect the same results. Disk isn't going away, no matter how hard some vendors wish.
Henry Newman, CTO of Instrumental Inc. and a regular Enterprise Storage Forum contributor, is an industry consultant with 28 years experience in high-performance computing and storage.
See more articles by Henry Newman.
Follow Enterprise Storage Forum on Twitter