Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
However, this future is anything but guaranteed. For this to come to pass, we must find good ways to effectively and efficiently move data from the spinning 7.2K drives to the SSDs. I don't think putting solid state storage on spinning disks as extra cache is the right way to do this. However, there are some pretty interesting storage systems that use SSDs as a cache.
The most obvious way to integrate these two types of storage (low-cost, large capacity spinning drives and SSDs) is to use tiering. The key for tiering to be successful is that it has to be able to move data quickly between the two tiers. For example, you could intercept open() calls from an application to start moving the data between tiers. If this is coupled with a list of applications that are to use the fast storage while running, then you get the full benefits of the fast storage (SSDs) without the full cost. For this to happen, the storage system and the tiering must recognize when data needs to be moved and can move it quickly.
Alternatively, before an application runs, the data could be moved from the slow storage to the fast storage. This is typically referred to as "staging" the data. This approach can be used to great effect in the HPC world because applications are typically run using a job scheduler (also called a resource manager) where the user would specify such things as the application, how it is to be run, how many processors are needed, how much memory, and even which files are to be used for input and which files are output files. The user then submits the job to the job scheduler, which decides when and where the application is to be run based on the requirements specified by the user. The job scheduler could then copy the input files from the slow storage to the faster storage in preparation for the application to be run. When the application is finished, the job scheduler can move the files from the faster storage to the slower storage.
The tiering approach is a bit nicer than the job scheduler approach because it happens in the background unbeknownst to the user (automagically, if you will). However, existing tiering software is not really capable of quickly recognizing when data is needed on faster storage and quickly moving it there so it is of limited value today. Much of the tiering software requires the storage study the usage pattern of the blocks of data within a file. After a period of time, it might move them to faster storage based on what is has measured (and perhaps predicted). So, what is really needed is much better tiering software than today's pretty bad option.
This is just a prognostication--I could easily be as wrong as I'm right. But I do think it bears some thought about the direction storage is headed. We're seeing workloads that are much more IOPS based than we previously thought, but at the same time we need lots of capacity. The key point is that we really need only high IOPS performance when an application is running. Consequently, we can make a portion of the storage a fairly small SSD-based system that has tremendous IOPS capability but a fairly small capacity at a reasonable price point. Then, we also create a very large but lower performance storage pool using the huge capacity and great price point offered by 7.2K drives. As a result, I think we'll see 15K drives disappearing from the market during the next few years.
The key to being successful is to marry the two storage pools with middleware that enables the data to be easily moved between the two. Tiering software is probably the best solution, but the current state of tiering is pretty bad. What is needed is tiering software that quickly determines when data must be moved from slower storage to really fast storage and back again. This has to happen in a way that the application performance does not suffer too much because of the data movement.
Current tiering software does not even come close to this, but I have hopes that vendors understand this and are working to fix the problem.
Jeff Layton is the Enterprise Technologist for HPC at Dell, Inc., and a regular writer of all things HPC and storage.