Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Much of the industry seems to have gotten the picture that doing all flash storage is not very practical except for small contained sets such as file system metadata or database index tables, given the cost of flash. Nor is all disk practical except for streaming I/O environments such as video playback or captures.
Of course I am thinking of larger environments with hundreds of TB (or greater) of storage, where storage costs are a big cost driver. Almost every week – if not more often – you hear of another hybrid storage product that will cache your data and seamlessly make it seem like you are running off of one hundred precent flash storage. Of course reality and marketing materials are often different
On the other hand I do believe that the world is going to move to hybrid storage because:
1. Flash costs will not drop below disk costs.
2. Flash write performance is not improving as fast as disk performance.
3. Flash volume cannot replace disk storage without hundreds of billions dollars in investment.
So if you believe that my assumption that data that is regularly accessed (aka hot data), will be migrated from spinning disk to flash and data that is not used very much (aka cold data) will be migrated from flash to spinning disk, what are going to be the challenges for different implementation of flash cached storage?
All of this is very interesting given the latest release on the drop in revenue for flash storage of 6.6%, which means that there is not likely going to be the hundreds of billions of dollars in investment to replace disk drives.
Here are what I think the challenges are:
1. How do you get data to the right tier of storage at the right time?
2. What is the right tier of storage and how do you know it?
3. What are the advantages and disadvantages of the competing designs methodologies?
First I think it is important to cover the two competing design approaches:
1. Caching on the disk drive
2. Caching at the storage controller
Disk Drive Caching Advantage and Disadvantages
Hybrid disk drives have been on the market for less than a year, and the initial design provided only read caching and did not support write caching. But over time I suspect that will change, along with changes to what disk drives have flash cache and the sizes of flash cache.
Having cache at the disk drive allows for the caching system to place the management and controls at the lowest layer. The storage controller does not have to understand the underlying block structure for each disk drive and each tier – the disk drive itself maintains that in its firmware.
Those are great advantages, but the other side of the coin are that things like data deduplication and compression are not likely as useful as a storage controller that breaks things up into blocks and spreads them across the set of disk drives. You are farther removed from the file, and the topology of the information you are accessing is more difficult to understand.
Storage Controller Caching Advantages and Disadvantages
The storage controller has more knowledge of data layout for files than a disk drive does. (It is too bad we never got ANSI T10 OSD working so we could understand the topology of files, but that that is another story.) The storage controller does not have perfect knowledge of data layout as in the POSIX world. Only the file system and volume manager have that knowledge, but it has far better knowledge than a disk drive.
The storage controller can most likely do a better job at deduplication and compression for data and for some data types that makes sense, but for others that are pre-compressed like video it does not. The problem is that the storage controller must understand the performance and topology of all of the tiers of storage and know what is where and why. You might have idle disks, 5400 RPM disks, 10K RPM disks and a variety of tiers of flash. The other thing is that to really do this well the storage controller needs to be very large to control lots of different storage tiers. Large generally means (at least in the past) expensive.
The data challenges are likely what you expect given our POSIX world. Even though there is a movement to object storage, scale out object storage file systems that do anything other than mirroring are not common place. Yes, there are products, but we still mostly live in a POSIX application environment, and though we might want that to change quickly, it will not.
Moving Data to Right Tier
With a POSIX file system, as stated, only the volume manager and file system know which disk drives the files are allocated on. For the most part they are only allocated on sequential block address if there is only one write going on at the same time.