Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
A revolution is underway in the way enterprise SSDs are used in the data center. This revolution was boosted by an announcement that was made with little fanfare earlier this month, and its repercussions for enterprise SSD users could be titanic.
The announcement in question detailed the new NVM Express over Fabrics specification which NVM Express, Inc. released in June. This is the organization which developed the NVM Express (NVMe) specification for accessing solid state storage such as enterprise SSDs connected to the server's fast memory bus over a (slower) PCI Express (PCIe) bus.
Slower is a comparative term: the PCIe bus is much, much faster than using SATA or even SAS connections. NVMe takes advantage of the parallelism of flash storage to lower the I/O overhead, improve performance and reduce latency.
The NVM Express over Fabrics specification deals with accessing NVMe storage devices such as enterprise SSDs over Ethernet, Fibre Channel, InfiniBand and other network fabrics.https://o1.qnsr.com/log/p.gif?;n=203;c=204650394;s=9477;x=7936;f=201801171506010;u=j;z=TIMESTAMP;a=20392931;e=i
"The NVM Express over Fabrics specification extends the benefits of NVM Express beyond rack-scale architectures to datacenter-wide Fabric architectures supporting thousands of solid state devices, where using a fabric as an attach point to the host is more appropriate than using PCI Express," the organization said in its announcement.
Enterprise SSD Storage Closer to the Server
So why is this announcement part of something revolutionary? The answer boils down to latency, and the problem of getting data from the place it's being stored to where it's needed – a server's processor – when it's needed.
The latency problem stems from the fact that ever since IT's year zero, processor performance has increased at a faster rate than storage performance. It's been compounded by the fact that data storage requirements have exploded, meaning that reliance on direct attached storage (DAS) within a server is no longer possible. That means data is getting stored further and further away from server processors. The introduction of all-flash arrays populated by high-performance enterprise SSDs in the last few years has helped to reduce the latency problem, but in no way can it be said to have solved it.
To understand why not, it's important to get a clear understanding of how different data storage media affects latency. Typically latency is measured in nano-, micro- or milliseconds, with processor L1 cache memory capable of providing a processor with data in half a nanosecond. That's one half of one thousand-millionth of a second, an amount of time so small as to be impossible to comprehend.
Latency Measured in Days for Enterprise SSDs
Speaking at the Flash Forward storage conference in London in June, Chris Mellor, a storage expert at The Register, proposed a simpler way to understand latency in different storage media. Rebasing L1 cache memory to have a latency of 1 second, he said DRAM access latency (which is of the order of 100 nanoseconds) becomes 3 minutes 20 seconds. Reading a fast server-attached NVMe enterprise SSD such as a Micron 9100 NVMe PCIe SSD involves a latency of 2 days, 18 hours and 40 minutes, while an IBM FlashSystem V9000 all-flash array read involves latency of 4 days, 15 hours, 7 minutes and 20 seconds.
But if you think those latency periods are long, then consider this: an EMC Isilon NAS access would involve latency of more than 115 days, and a typical DAS disk access would take more than 6 years. And when it comes to SAN storage, the latency times are far, far, worse: a typical SAN array access involves a latency wait of a staggering 19 years, 5 days, 10 hours and 40 minutes.
So here's the problem: today's servers have super-powerful processors with multiple cores, but there's no way to move the data to the processors fast enough to keep them busy. You can upgrade your storage systems to all-flash arrays with as high-speed enterprise SSDs if you like, but if the rest of the infrastructure can't cope then what, asks Mellor, is the point? "It's like taking a 20-minute flight across the country, and then spending four hours driving to the office from the airport."
The obvious solution is to put all your data in the processor's L1 cache (or better still the processor's registers), but sadly that's not an option. For one thing, the heat generated by anything but a tiny cache would be enough to melt the processor. Another partial solution is to use more fast, expensive DRAM, but DRAM is volatile and space-limited. So if you want to keep your data as close as possible to the processor that leaves solutions like NVDIMM or PCIe enterprise SSDs within the server.
These solutions are also expensive, and space is constrained. More importantly, these solutions don't deal with the need for shared storage.
Enterprise SSDs Connected Over Fabrics
And that's why NVM Express over Fabrics is so exciting. It means you can take an NVMe enterprise SSD and take it out of the server. By connecting it using Ethernet or InfiniBand, you can treat reads like remote direct memory access (RDMA), said Mellor. "It's like a memory-to-memory transfer, and at 100 nanoseconds its nearly the same as a local direct flash access," he said. "If you have that, then instead of waiting 19 years you'd wait a couple of hours," he added. "NVMe fabrics are a great way of bringing storage nearer to the processor. They're revolutionary."
Systems built on the new NVM Express over Fabrics specs will receive a huge performance boost when the NVMe enterprise SSDs they contain are upgraded with next-generation solid state storage media to replace the existing NAND storage. Although there has been talk of HPE Memristor and IBM phase change memory (PCM) products, it's likely that the only next-generation medium to emerge in the near future will be Intel and Micron's 3D XPoint.
3D XPoint and Enterprise SSDs
Initially, it's likely that 3D XPoint will be used as a lower-cost substitute for DRAM, blurring the distinction between memory and storage, according to Jim Handy, solid state storage expert and semiconductor analyst at Objective Analysis. Intel plans to release 3D XPoint-based DIMMs towards the end of this summer, while its 3D XPoint Optane NVMe enterprise SSDs won't be available until later in the year. (Micron has yet to release any information about its plans for the technology.)
When 3D XPoint-based enterprise SSDs do finally become available, this will increasingly blur a different boundary: the one between internal and external storage offerings. Fast NVM Express over Fabric storage will effectively work as shared DAS, offering slightly faster performance than SAS SSD DAS (and slightly slower than PCIe NVMe enterprise SSDs) even though the enterprise SSDs themselves will be physically located outside the server.
In fact, the NVM Express over Fabrics revolution started to happen even before the publication of the NVM Express over Fabrics specs. Offerings that make use of conventional enterprise SSD flash storage are already available from Apeiron, EMC DSSD, E8 and Mangstor Technology, with another from Tegile promised in the near future. The NVMe enterprise SSDs they require are readily available from vendors including Samsung, HGST, Intel, Toshiba, Micron and OCZ.
So, looking to the future, what will be the effect of this new style of low-latency, close-to-the-server, shared storage?
Mellor predicts nothing less than an I/O bound app revolution. In particular, the combination of NVM Express over Fabrics and Hadoop will result in very fast Big Data analytics. That's because instead of having to move data from Hadoop node to Hadoop node for processing, vast quantities of data will just sit in shared DAS storage — accessible over the fabric by any processor with minimal latency. It might also result in archive systems that are priced between power-down disks and tape, but offering instant access to the archived data.
All in all it's an exciting time for anyone involved in solid state storage. Expect to hear much more about NVMe enterprise SSDs accessed over fabrics. The publication of the specs is just the end of the beginning of developments.
Photo courtesy of Shutterstock.