Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Five years ago I wrote an article about the coming wall for flash chip density. Increasing flash density is important because it allows manufacturers to put more capacity in a given amount of space. It should also decrease costs which has been the Achilles heel of flash storage for a long time.
You can achieve increased flash density a couple of ways. One way is to decrease the feature size of the flash chips. A second way is to change the technology. You could even argue that there is a third way which is to use both approaches. Regardless, the desire is to improve the price of flash drives while maintaining or improving performance and increasing drive capacity.
A great deal has happened since I wrote that article discussing why flash chip density would stop growing because of feature size limitations. Nonetheless, the price/capacity ratio ($/GB) has greatly improved. But how is this possible because of the chip density limitations? Did chips actually increase in density or were other factors involved?
Reducing Chip Feature Size
As I discussed in my original article, one way to improve flash chip density is to reduce the feature size on the chips. It appears that generally flash drive manufacturers are doing mass production around 20nm (plus or minus). Most of them have demonstrated flash chips smaller than 20nm, but they have not really appeared in quantity (Note: I view the recent Plextor drive at 19nm as virtually the same as 20nm.) You can get smaller than 20nm, but you will have to use a great deal more error correcting because of the small feature size, which leads to a greater possibility of data disturbance due to a write. You may also have to tolerate a lower Program/Erase (P/E) number, and to maintain capacity, more over-provisioned space may be necessary. The combination of these factors offsets the benefits of the smaller feature size and are likely to result in flash drive prices flattening.
Yet flash chip manufacturers have continued to drive down prices and even improve performance. Since reducing the feature size of flash chips is not a good way to reduce drive prices, flash manufacturers have focused on other techniques for making denser flash chips and reducing the cost.
Denser cells - TLC and more
The initial flash chips used Single-Level Cell (SLC) technology, where each cell can hold one bit of information (two states). These chips are fast, use relatively little power, have long endurance (lots of P/E cycles), but have low density, leading to less dense and more costly drives. Multi-Level Cell (MLC) chips use cells that can store two bits of information (four states) per cell. This doubles the chip density at the cost of a slower write speed, slightly higher power and about one-tenth of the endurance (number of P/E cycles) relative to SLC. MLC increases the density of the flash chips but lowers the cost.
Fairly recently, Triple Level Cells (TLC) have been introduced. These cells can store three bits of information (eight states) each. This triples the density relative to SLC, which can really reduce the cost of flash drives. But these chips have slower write speeds compared to SLC and MLC. They also use slightly more power than MLC and have about one-tenth the endurance of MLC chips. The table below lists rule of thumb endurance and performance numbers for the three types of flash cells.
|Bits per cell||1||2||3|
As you can see from the table, the performance and endurance of the cells decreases as you add the ability to store more bits. However, the reason more bits/cell is so attractive is density. Using the same feature size (such as 20nm), a 16 GB SLC chip becomes a 32 GB chip using MLC, and a 48 GB chip using TLC. This reduces the cost of the chips on a per gigabyte basis, which means more capacity. So more people buy the drives, which can further lower the price, and so on.
To help with the low endurance, TLC manufacturers have increased over-provisioning (the chips are cheaper after all) and improved the controller's wear-leveling. They have also added more error correcting capability. The result is a much more even wear on the chips as well as a pool of "spare" cells that can be used if needed (reallocation). There is an article from 2012 that tested the endurance of a TLC drive, the Samsung 840. For client workloads TLC reduced the total estimated lifespan by about 2/3 relative to MLC, but for the smaller drive, 128GiB, the expected life was still almost 12 years. Enterprise workloads are sometimes more write intensive than client workloads, reducing the lifespan. But don't be surprised if the combination of over-provisioning and wear leveling keep TLC chips competitive.
Recently there was a long-term test of SSDs to see how much data could be written to the drives before they died. They tested consumer drives all with a capacity of around 250 GB (plus or minus). The goal of the test was to determine how much data can actually be written to each of the drives before they die. The test should stress both the flash chips and the controllers. One of the drives was the 240GB Samsung 840 which uses TLC flash chips. They also tested the 250GB Samsung 840 Pro which uses MLC chips. The other drives used MLC and included two drives with a SandForce controller. One of the SandForce drives used incompressible data, negating the compressing feature of the controller. The other drive used 46 percent incompressible data allowing the controller to do some compression. This allows the drive to perhaps write less data than the drive with incompressible data.
The first two casualties of the testing were actually MLC based SSDs. The tests showed that around 700TB, the Samsung 840 was using its over-provisioned cells at a steady rate but was still functioning. The 840 started logging reallocated sectors after around 100TB of data written and started steadily reporting them after that point. It also started allocating more spare area to replace bad blocks, as one would expect with a TLC based drive. It was still functioning after 900TB but died shortly thereafter, not quite making one Petabyte. In comparison, the Samsung 840 Pro, which uses MLC, managed to write just a little over 2.4PB of data before failing (one of the last drives standing).
There are a number of TLC drives available for the consumer market. A quick glance at Newegg on 4/5/2015 showed that TLC drives were about $0.50/GB for the range of 250GB to 1TB in capacity. The MLC drives start at just under $0.50/GB at 120TB but decrease to about $0.35/GB at 240TB. For drives up to 1TB the $/GB rises slightly but is still under $0.40/GB.https://o1.qnsr.com/log/p.gif?;n=203;c=204650394;s=9477;x=7936;f=201801171506010;u=j;z=TIMESTAMP;a=20392931;e=i
Even though MLC drives are a touch less expensive than TLC drives, it is predicted that TLC chips will start to dominate the flash market in 2015, as shown in Figure 1 below.
[Source: http://www.thessdreview.com/daily-news/latest-buzz/silicon-motion-announces-sm2256-worlds-first-6gbs-ssd-controller-supporting-tlc-nand/ (used with permission)]
By the end of 2015 it is predicted that TLC will make up 50 percent of the flash chip market. But also notice that the market share for SLC chips also picks up to about 10 percent at that time.
TLC is even making its way into the enterprise world with the launch of the Samsung 845DC EVO. When it was released, it was about $1/GB. However, now you can get the 960TB Samsung 845DC EVO drive for about $660 ($0.69/GB).
The original flash chips were planar. That is, the cells were all on the same level within the chip. However, companies realized that they could increase the chip density by going vertical. This led to the development of 3D flash chips. These chips are now available and are helping to increase the density and drive down the $/GB.
As with TLC, Samsung is the most aggressive company in terms of bringing 3D NAND to market. As the chart in Figure 2 illustrates, 2015 will be the year that 3D flash drives will appear from a number of vendors.
[Source: http://www.theregister.co.uk/2014/12/16/3d_nand_flash_of_the_future/ (used with permission)]
As the chart explains, Samsung is already shipping drives that use 32 layers of MLC cells in the flash chips. These are the Samsung 850 PRO and 850 EVO. The Samsung 850 Pro has a slightly higher price point than the Samsung 850 EVO, as shown below in Table 2 (prices taken from Newegg.com on 4/5/2015).
|Samsung 850 Pro||128GB||$96.00||$0.75/GB|
|Samsung 850 Pro||256GB||$159.00||$0.62/GB|
|Samsung 850 Pro||512GB||$290.00||$0.566/GB|
|Samsung 850 Pro||1TB (1,024GB)||$552.00||$0.537/GB|
|Samsung 850 EVO||120GB||$73.00||$0.608/GB|
|Samsung 850 EVO||250B||$110.00||$0.44/GB|
|Samsung 850 EVO||500GB||$210.00||$0.42/GB|
|Smasung 850 EVO||1TB (1,024GB)||$390.00||$0.381/GB|
As you can see, 3D MLC NAND is currently priced very close to MLC drives, and they are expected to come down in price.
While Samsung is the first out of the gate with 3D NAND chips, they definitely won't be the only one. The Intel-Micron joint effort announced that in the second half of 2015 they will have a 256 Gbit (32GB) capacity die that uses 32 layers (MLC) [Note: Samsung is currently using a 120 Gbit die]. They will also have a 384 Gbit (48GB) die that uses TLC, increasing the flash chip density by 50 percent.
Recently, Toshiba announced a 48-layer 128Gbit (16GB) 3D NAND chip. They are sampling these chips now, but the speculation is that they will ship a drive in 2016.
The advent of 3D NAND chips has just started to impact the flash drive market, but it is expected to have a significant impact in the coming two to three years. If you couple this with TLC chips, then you can see the flash chip density is likely to increase quickly with the result that the flash drive prices should decrease.
As vendors started to develop flash drives they soon realized that drives were throttled by the current storage buses, SATA and SAS. In addition to dropping the price of flash chips, vendors started looking for ways to improve performance by going around SATA and SAS.
Ever since Fusion-io introduced SSDs that attached using the PCIe bus, people have realized that the SATA and SAS buses are definitely inhibiting performance. On the one hand, SATA and SAS buses allow drives to be easily connected and used as IO devices. All of the existing software infrastructure could be used with these buses. On the other hand, using them holds back performance. Therefore, vendors have started to examine using the PCIe bus to really take advantage of the performance of these drives.
In addition to the PCIe bus, the new SATA Express interface offers improved performance options for consumer drives. It supports both SATA and PCIe storage devices. In particular, the M.2 specification for internally mounted expansion cards offers the option of physically small SSD devices with good performance.
As mentioned previously, PCIe flash drives have been around for a while, starting with Fusion-IO. The current prices have been a little high and vary by a fair amount. As of 4/5/2015, consumer PCIe drives started around $0.67/GB and went to $5.00/GB with a typical price range of $1.00/GB to $3.00/GB. These are 2x to almost 20x the price of SATA or SAS drives, but the trade-off is that you get much greater performance. For example, for a high-end drive, you can get up to 2.8 GB/s for reads and writes, and about 410,000 random IOPS for reads and writes. The typical performance is a bit less than this, but it illustrates what kind of performance PCIe offers.
The early PCIe SSD drives all had their own drivers, which made life a little difficult. The recent NVMe(Non-Volatile Memory Express) specification was developed to standardize on a logical device interface so operating systems only need one driver. Most importantly, the specification takes into account the low-latency nature of PCIe SSDs and the parallelism of current CPUs and PCIe SSDs. The interface applies to both PCIe devices and SATAe/M.2 devices.
There have been NVMe drives for a bit, but an example of a recent drive is the Intel SSD 750. It's not quite available as of the time of this writing, but the preliminary performance is quite good. The smaller drive, 400GB, can reach 2.2 GB/s reads, 900 MB/s writes, 430K random read IOPS, and 230K random write IOPS. The price for the smaller driver and the larger 1.2TB drive are both about $1.00/GB. This illustrate the extreme performance from NVMe drives and the rapidly decreasing prices.
M.2 drives are also becoming quite prevalent. They are physically very small, but then again, flash chips are small. Despite this, they currently range in size up to 1TB. For example, Samsung has the 840 EVO mSATA drive that uses TLC chips. Also, recently, they have introduced a 3D NAND M.2 drive, the M.2 850 EVO that uses TLC as well. Both drives are capable of about 520-540 MB/s for reads and writes and about 88-95K of random IOPS (read and write).
For a period of time, it didn't look like flash drives were going to decrease in price very much. Flash cell technology is limited to around 20nm because of cost and complexity considerations, but manufacturers have found ways around the limitation. Rather than decrease the features size, they now store more bits per cell (TLC) and have started to create 3D flash chips. This combination, plus the growth in flash storage sales, has driven down the price per gigabyte.
Flash drive vendors aren't totally focused on price though. They have also been working on ways to take advantage of the flash drives by utilizing other system buses, specifically the PCIe bus. The somewhat new NVMe specification has standardized PCIe storage devices, reducing the burden on writing drivers and adding more headroom on performance via parallelization. When flash storage is put in the PCIe bus, you can see amazing performance from single drives.
These next couple of years with flash drives are going to be exciting.
Photo courtesy of Shutterstock.