Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
A number of trends are coming together that are likely going to change how we use and access data storage, probably near the end of the decade.
At the Intel Developers Forum last year, there was an interesting talk by Alan Gara, Intel’s Chief Exascale Architect. Here’s a more detailed article on Intel's plans and what Intel says about faster optical interconnects and the issues with power, CPU, memory density and performance challenges. It’s definitely worth the time to listen and read the links.
Combine those predicted shifts with what has been happening with the storage industry with changes to interconnects, then add in what has been happening to the base storage technology, that is, the storage device themselves (disk, flash and tape). Clearly, change is coming. We have been in the same paradigm for a few decades: CPU, connected to memory and with connections to external channels for the display, and communications.
For an even longer period of time, disk drives have been attached over channels during the last few decades, from proprietary channels to IPI-2 and IPI-3 to SCSI, fibre channel and now SAS. Things are going to have to change for us to move forward and I think one of the first changes will come from Intel, based on the developments noted above.https://o1.qnsr.com/log/p.gif?;n=203;c=204650394;s=9477;x=7936;f=201801171506010;u=j;z=TIMESTAMP;a=20392931;e=i
So what happens to ethernet if Intel decides to put communications on the socket? What happens to storage communication? What happens to disk drive interfaces or, you might ask, what happens to SAS, since Seagate now makes an ethernet attached object disk?
Each of these areas has impact on the other area. Seagate, Toshiba and WD do not make CPUs and ARM. AMD and Intel do not make disk drives, but there are lots of overlapping issues between these two technologies. And as was discussed in the Intel talk, memory is going to be a big challenge.
For example, most if not all storage controllers today use standard commodity CPUs for much of the controllers’ RAID activities. Most of the controllers use Intel CPUs because of the desire for PCIe 3 and the performance.
Let me predict what I think is going to happen by the end of the decade. But please do not hold me to this, as a great deal depends on the companies involved and which vendor moves the fastest – and all of that is nearly impossible to predict.
I think all the CPU vendors have a plan to move more communication directly on the socket. There has been a significant evolution over the last decade or so with more and more technology being moved to the socket. Today we have USB, PCIe 3, ethernet and other features right on the CPU for servers. And for cellphones we have other communications connections and requirements for things like analog to digital conversion and the like.
The other big requirements for CPUs are memory and the connections to memory. Today for servers we have L1, L2 caches and in some cases L3, and then standard DDR memory. I see much of this changing in the future with more hierarchies of memory, especially beyond DDR where there will likely be a non-volatile area of memory.
If you listened to the whole Intel video you will note that memory is one of the biggest issues as CPUs get faster over time and power will continue to have a profound impact on the design of systems. The SOC (system on a chip) concept is going to be even more system oriented in the future. Intel has been pretty clear with their interconnect plans, having purchased Fulcrum, QLogic IB division and Cray’s networking division. The ARM consortium is working on new connections and of course everyone has heard about the OpenPower group that IBM, Mellanox and others have joined.
Clearly there is a move afoot to change the interconnect that the processors in use. Does that this mean that PCIe and PCIe connectivity is dead? Honestly I think we are all in for a big surprise in this area. I think we are going to back to a proprietary world with a few small changes. Back in the 1980s and 1990s, DEC, IBM, Sun and others had their own buses and connectivity. But the vendors found that the broad market was too small to support the cost for the NIC and HBA, and other vendors had to support these server vendor’s products.
Along comes PCI and – shazam – the NIC, HBA and other vendors were able to dictate in many cases that they were not going to build products for these vendors without money up front or guaranteed revenue. They could do this as other vendors had moved to PCI and the market was large enough and easy enough, as these vendors had a standard interface for peripheral devices.
The world has changed a quite bit and now the CPU vendors control much, if not all, of the board and there are not many CPU vendors left. The CPU vendors can tell the peripheral vendors that they need to play ball with them rather than the other way around, if the CPU vendors decide to build new connectivity options.
With ethernet technology being built right on the CPU socket by a number of the CPU vendors, and others adding SATA or SAS, you might realize as I have that the CPU vendors are trying to cover the a majority of the normal connectivity options. They’re cutting out the peripheral vendors from at least some of their market share. This in itself is a pretty big market change and there is more to come as far as I can tell.
The Storage Side
On the storage side I think we are also seeing other signs of big changes ahead. Going back to the 1970s and 1980s we had many different types of drive interfaces and a large number of drive vendors. Today we have three vendors, basically Seagate, Toshiba and Western Digital, and we have a few SSD vendors that make devices such as Intel and SanDisk. And of course we have vendors making PCIe SSDs and with the new NVM Express standard some are saying that we do not need SAS or SATA anymore. This might work for the low end where there are one or at most a few storage devices, but this is not going to work for systems with many hundreds of terabytes of storage.
NVM Express is going to work very well in laptops and other devices. For example, my travel laptop uses an Intel Atom processor and a SSD. Intel could use NVM Express and would not have to use SATA for the communication on the chip, saving the space for something else on the chip that was not communication to storage, maybe external communication, for example.
The point is that on the low end NVM Express might have a significant impact on the market and change storage connectivity, which has not changed much in over two decades. The storage vendors are not taking the changes by the CPU vendors without their own consolidation. WD purchase a number of SSD companies over the last few years, and I think it has much to do with the changes in the interfaces that are coming over the next few years.
Seagate is taking a different tact I think, with object disk drives, which is addressing a completely different part of the market than consumer storage. Object disk drives are for the cloud market, not the consumer market, and this is a completely different strategy than what WD is doing. Two out of the three major disk drive makers (and it should be noted that Toshiba is a very small player compared to WD and Seagate) are making a move to enter new markets in completely different ways.
With SATA lagging behind SAS communication performance (SATA at 6 Gbit/sec and SAS now at 12 Gbit/sec), and with SATA not able to keep up (next generation SATA will be 8 Gbit/sec), this means that a new interface is likely going to have to emerge at the low end. Or SAS, which is considered an enterprise interface, is going to have to move down market. I honestly do not see either happening quickly.
So What Does it Mean for Data Storage?
If we want to continue to thrive in our industry we better get ready for some significant changes coming down from the market to our computer rooms. I think we are going to see the following changes:
1. We are going to see a move for low power processors for desktop applications. This is going to happen with all processor types, from ARM to Intel. And for all we know we might see another processor type emerge, and it would not surprise me if it came from China.
2. The low power processor will likely have non-volatile storage connected directly off of some type of bus and will not use SATA or SAS protocol.At least for Intel this will be NVM Express but for ARM and others it will be something different and it is unclear, at least publicly as of now, what that will be.
3. Storage controllers as we know them are going to change. Back in the mid-2000s I was a strong proponent of T10 Object Storage Device (OSD), but this technology never made it through the recession and there are no T10 OSD disk products in the market. During the same time period the REST interface has taken off like wild fire and is being used from the desktop to the enterprise. Long term I think that REST will become the dominant force in the market and block-based storage (file system based) and file-based (NAS) storage is going to be replaced. The only exception might be for block-based storage, some very specific environment that needs the direct access and high performance that can only be done via the POSIX interface of OPEN/READ/WRITE.
The way this all plays out is that, I think by the end of the decade the CPU vendors take over the motherboard and the communications sector, putting pressure on Cisco and the other networking and storage communications vendors. And the storage controller vendors are going to be pressured from below by the disk drive vendors.
All of this is going to make for some interesting fights in our industry over the next 5 years and we should be preparing for some large changes. But all of this could change based on who make the right moves and who makes some wrong moves. Stay tuned.
Photo courtesy of Shutterstock.