What Is NVMe?

Short for Nonvolatile Memory Express, NVMe is a protocol that makes solid state drives (SSDs) faster, and it’s becoming increasingly popular among enterprise users.

The easiest way to understand what NVMe does might be with an analogy. Imagine that you just bought a sports car that can go 15 times faster than a typical car. Instead of topping out around 100 mph, your new car can go 1,500 mph.

The only catch is that to get to and from the freeway, you have to go down side streets with 25 mph speed limits and bumper-to-bumper traffic.

You’ll know that your entire commute could be even faster if only you didn’t have to waste time on the stupid side streets. And you’ll probably start looking for faster alternatives.

This scenario is a little bit like the situation in the storage industry after manufacturers introduced flash-based SSDs. Flash technology is many times faster than the spinning disks in a hard drive (how many times faster depends on which devices you are using), but the early drives all connected to systems and networks using the same SATA or SAS connections that hard drives do. While these interfaces are more than adequate for the type of performance hard disk drives (HDDs) can provide, they created bottlenecks for SSDs. It was particularly a problem for enterprise big data analytics and other data-intensive workloads.

Those performance problems spurred the search for better ways to connect SSDs to servers, and that’s where NVMe comes in.

Image source: ATP

What is NVMe?

The NVMe protocol standards are overseen by NVM Express Inc., a consortium of more than 100 organizations interested in developing faster protocols to improve the performance of nonvolatile memory. The organization is lead by a board of 13 promoters, which includes Cavium, Cisco, Dell EMC, Facebook, Intel, Micron, Microsemi, Microsoft, NetApp, Samsung, Seagate, Toshiba Memory and Western Digital.

The official definition of NVMe describes it as “an open collection of standards and information to fully expose the benefits of nonvolatile memory in all types of computing environments from mobile to data center. NVMe is designed from the ground up to deliver high bandwidth and low latency storage access for current and future NVM technologies.”

NVMe is an interface specification for connecting storage to servers via the PCI Express bus. In layman’s terms, it is a faster way for SSDs to communicate with their host systems. It helps alleviate the bottleneck that occurred when very fast flash was connected to systems via the SAS or SATA connections that were originally designed for HDDs.

The Intel 750 series is among the many solid state drives that incorporate NVMe technology.

Image source: Intel

NVMe speed is much faster than SATA or SAS

An NVMe controller speeds up performance in a couple of different ways. One is that is uses the PCIe bus, which connects the storage directly to the system CPU. This direct connection eliminates some of the steps necessary with SATA and boosts overall performance.

In addition, NVMe SSDs implement parallelism on a grand scale to improve throughput dramatically. As data is being transferred from the storage to the server host, it gets in a line, or a queue. Legacy SATA connections can support just one queue and only 32 pieces of data can get in line at a time. Going back to that car analogy again, it’s like having just one lane of traffic with room for 32 cars on the road.

But NVMe storage supports up to 64,000 queues with 64,000 entries each. In other words, it’s like going from a one-lane road to a 64,000-lane road with room for 64,000 cars in each lane. And as the chart below illustrates, that makes a huge difference in overall performance.

Remember back to our car analogy and the 25 mph speed limit on the side streets? SATA and SAS connections impose a speed limit as well. For SATA, the theoretical maximum transfer speed is 6.0 Gbps (and in practice, the maximum transfer speed is much lower). That puts an upper limit on how fast SATA SSDs can go. Beyond a certain point, using faster flash has no impact on the overall performance of the system because the SATA connection creates a bottleneck.

For most consumers, SATA SSDs provide more than enough performance for the tasks they are likely to do on a regular basis. And because SATA SSDs are less expensive than NVMe SSDs, most consumer solid state storage still uses that SATA interface.

However, enterprises are generally working with a lot more data than consumers, and for them, the latency imposed by the SATA connection can become an issue.

NVMe offers a fast faster means of moving data through the system than SAS, hence its use in today’s cutting edge systems.

Image source: PureStorage

Can only SSDs run on NVMe?

Because NVMe drives are so much faster than SATA, it seems like it would be a good idea to use NVMe with hard drives to give them a boost in performance. Remember, however, that NVMe stands for Nonvolatile Memory Express, and it was designed specifically to work with nonvolatile memory like Nand flash (although it can also be used with newer forms of nonvolatile memory, like 3D XPoint.)

When a system is reading data off a spinning disk, it can read only one piece of data at a time. That’s because it has to spin around until it gets to the correct physical location for the first piece of data, and then spin around and move to the right location for the second piece of data, and so on.

Flash and other nonvolatile memory technologies, on the other hand, have no moving parts. That means the system is free to read data from many different locations at the same time. And that’s why SSDs can take advantage of the parallelism offered by NVMe, but HDDs cannot.

PCIe and NVMe are related, but they’re not the same thing

For many people, the most confusing part of NVMe is its relationship to PCIe. Some vendors refer to their SSDs with the NVMe label, and some use the PCIe label, and some seem to use the terms interchangeably.

While PCIe and NVMe are very closely related, the terms refer to slightly different technologies. You can think of PCIe as being a physical part of your system. When you plug an NVMe SSD into your server, you will connect it via a PCIe slot.

NVMe, by contrast, is a protocol, a group of hardware and software standards that allows the SSD to use the PCIe bus. From a layman’s perspective, you can think of NVMe as being the language that allows the storage device to connect with the server, while PCIe is the actual physical connection.

NVMe over Fabric connects SSDs to networks

Most of this article has focused on standard NVMe, which connects SSDs directly to servers, but the NVM Express organization has also published a specification for NVMe over Fabric (NVMe-oF), which connects nonvolatile memory used for block storage to a network. According to the group, “NVM Express over Fabrics defines a common architecture that supports a range of storage networking fabrics for NVMe block storage protocol over a storage networking fabric. This includes enabling a front-side interface into storage systems, scaling out to large numbers of NVMe devices and extending the distance within a datacenter over which NVMe devices and NVMe subsystems can be accessed.”

About 90 percent of the NVME-oF specification is the same as the NVMe specification; however, it does use a different transport-mapping mechanism. There are also two different variations of NVME-oF: one for RDMA and one for Fibre Channel.

The NVMe-oF specification also much newer than the NVMe specification, and while some vendors have announced support for the technology, few are actually selling NVME-oF products. Look for this technology to become more widespread in coming years.