Fixing SSD Performance Degradation, Part 2 Page 4 - EnterpriseStorageForum.com

Fixing SSD Performance Degradation, Part 2 Page 4

All of the results are presented in bar chart form with the average values plotted and the standard deviation shown as error bars. For each test there are two bars plotted - "before" and "after". The "before" results are the initial benchmarks of the SSD before it was stressed by using IOR. The "after" results are the benchmark results after the device was stressed. We are looking for noticeable differences in the "before" and "after" results.

There will be three sections of results: (1) Throughput (performed using IOzone), (2) IOPS (also performed using IOzone), and (3) Metadata (performed using metarates).

Throughput Results

The first throughput result is for the write throughput test. Figure 2 below presents the results.


Figure 2 - Write Throughput Results from IOzone in KB/s



You can see that for three of the record sizes, the write throughput performance was slightly better after the I/O intensive tests were run (average values). However, you have to pay very close attention to the standard deviation. If the differences of the two tests falls within the standard deviations then it is impossible to say that "after" is better than "before."

 

For example, for a record size of 1MB, one might think that the "after" results are better than the "before" results. However, the difference between the two tests falls inside the standard deviation, making it impossible to say with reasonably certainty that "after" is faster than "before".

Figure 3 below presents the results for the second test - the re-write throughput results.


Figure 3 - Re-write Throughput Results from IOzone in KB/s



For the re-write throughput benchmark, at a 4MB record size, it appears that the "before" results are a bit faster than the "after" results. The difference is fairly small however - about 3.5%. For the other record sizes, the re-write performance difference between before the stress testing and after, fall in the standard deviation or are so small that the difference is really minor.

 

Figure 4 below presents the read throughput results.


Figure 4 - Read Throughput Results from IOzone in KB/s



Despite the appearance that the "before" results are better than the "after" results, the differences are within the standard deviation so it is impossible to conclusively say that "before" is better than "after" for this test.

 

Figure 5 below presents the re-read throughput results.


Figure 5 - Re-Read Throughput Results from IOzone in KB/s



For this test, we don't see much difference in performance between the "before" and "after" benchmarks.

 

Figure 6 below presents the random read throughput results.


Figure 6 - Random Read Throughput Results from IOzone in KB/s



In this test we actually see quite a bit of difference between the tests. The "before" performance is much better than the "after" performance. For record sizes of 4MB, 8MB, and 16MB, the difference is about 13-15% in favor of the "before" results (this is average performance).

 

Figure 7 below presents the random write throughput results.


Figure 7 - Random Write Throughput Results from IOzone in KB/s



These test results are a bit different than the random read tests. At a record size of 4MB, the performance before the stress test ("before") was quite a bit better than "after" the stress test (a bit over 20%). But at the other record sizes the "before" and "after" performance appear to be within the standard deviation of each other. Consequently, it's impossible to say that one is better than the other.

 

Figure 8 below presents the backwards read throughput results.


Figure 8 - Backwards Read Throughput Results from IOzone in KB/s



For the three largest record sizes the performance prior to the stress test is much better than after. The difference is about 17-20%. However, this I/O pattern is rather unique to a small set of applications but you do see this pattern in some important HPC applications.

 

Figure 9 below presents the record rewrite throughput results.


Figure 9 - Record Rewrite Throughput Results from IOzone in KB/s



The performance before the stress testing and after are about the same for this particular benchmark.

 

Figure 10 below presents the strided read throughput results.


Figure 10 - Strided Read Throughput Results from IOzone in KB/s



For this particular I/O pattern we see that the performance prior to the stress test is quite good compared to after, particularly for the largest three record sizes. In general the "before" results are about 22-23% better than the "after" results.

 

Figure 11 below presents the fwrite throughput results.


Figure 11 - Fwrite Throughput Results from IOzone in KB/s



For this test there is very little difference between the performance before or after the stress test.

 

Figure 12 below presents the frewrite throughput results.


Figure 12 - Frewrite Throughput Results from IOzone in KB/s



As with the fwrite results there is very little difference between the performance before or after the stress test.

 

Figure 13 below presents the fread throughput results.


Figure 13 - Fread Throughput Results from IOzone in KB/s



There is some difference in the averages for the the "before" and "after" performance but they are within the standard deviation of the results so it is impossible to say that one is better than the other.

 

Figure 14 below presents the freread throughput results.


Figure 14 - Freread Throughput Results from IOzone in KB/s



In this last throughput test we do see some difference in the performance before the stress testing as compared to after the testing. The differences in the average performance at the large record sizes (4MB, 8MB, and 16MB) range from about 9% to a little over 13% better for the "before" performance compared to the "after" performance.

 

Overall we see some differences in the throughput benchmarking. The differences favor the "before" performance for certain tests:

  • Random Read (13-15%)
  • Random Write (20% for 4MB records)
  • Backwards Read (17-20% for larger record sizes)
  • Strided Read: (22-23% for larger record sizes)
  • freread: (9-13% for larger record sizes)

The random I/O performance difference is very interesting because when many clients applications access the same storage the I/O patterns basically become random to the storage device. Consequently, random performance of the device becomes much more important.

 

IOPS Results

Recall that four IOPS tests were performed: Write IOPS, Read IOPS, Random Write IOPS, and Random Read IOPS. The results for these four tests is presented in the following four figures.

Figure 15 below presents the Write IOPS results.


Figure 15 - Write IOPS Results from IOzone in Operations per Second



There is little difference in Write IOPS before and after the stress testing. There might be a small difference at a record size of 8KB, but the standard deviations are very close to each other so the difference in performance is very, very small.

 

Figure 16 below presents the Read IOPS results.


Figure 16 - Read IOPS Results from IOzone in Operations per Second



There is very little difference between the performance before the stress test and after.

 

Figure 17 below presents the Random Write IOPS results.


Figure 17 - Random Write IOPS Results from IOzone in Operations per Second



The only real difference in performance between "before" and "after" comes for larger record sizes. For 32KB and 64KB there is some difference with the "before" performance being slightly better. At a record size of 32KB, the "before" performance is about 10.8% better than the "after" performance and at 64KB, the "before" performance is about 8.7% better.

 

Figure 18 below presents the Random Read IOPS results.


Figure 18 - Random Read IOPS Results from IOzone in Operations per Second



This benchmark result is a little more interesting in that at the larger record sizes, the Random Read IOPS performance is actually better "after" the stress testing than "before". At 32KB the "after" performance is 14.26% better than the "before" performance and at 64KB the "after" performance is 12.6% beter than the "before" performance.

 

Overall the difference in IOPS performance between "before" and "after" the stress testing is very, very small. However, we do see some differences in the random IOPS performance between the "before" and "after" benchmarks.

Metadata Results

Recall that there are three metadata results to be presented: (1) File Open/Close performance, (2) File Stat performance, and (3) File Utime performance. Figures 19, 20, and 21 below, present the performance for the "before" and "after" benchmarks.

Figure 19 below presents the Metadata File Create/Close results.


Figure 19 - Metadata File Create/Close Results from Metarates in Operations per Second



The only real difference in performance is with 4 processes (NP=4) where the "before" performance is better (9%) than the "after" performance.

 

Figure 20 below presents the Metadata File Stat results.


Figure 20 - Metadata File Stat Results from Metarates in Operations per Second



There is some difference in performance between the before and after testing with the "after" performance being 3.88% better at NP=1, 6.58% at NP=2, and 3.44% at NP=4.

 

Figure 21 below presents the Metadata File Utime results.


Figure 21 - Metadata File Utime Results from Metarates in Operations per Second



There are some differences in the average performance between the before and after testing for this benchmark. But the difference lies within the standard deviation so it's impossible to say that one is better than the other.

 

Interestingly, there are some performance differences in metadata performance, but they indicate that the "after" performance is better than the "before" performance.

Summary

The intent of this benchmarking exercise is to test an enterprise class SSD with a recent Linux kernel version that supports TRIM, and look for performance degradation between a fresh SSD and one that has been through some I/O stress testing.

To compare the performance before the stress testing to the performance after the stress testing, three classes of benchmarks were used - (1) Throughput testing, (2) IOPS testing, and (3) metadata testing. These benchmarks were first run on a clean fresh drive and then on the same drive after it had been subjected to some fairly intense I/O workloads.

An Intel X25-E drive was used for the testing, along with a CentOS 5.4 system with a 2.6.34 kernel that has been patched with bcache. Ext4 was used as the file system because it supports the TRIM command. The drive was configured for the best performance possible on an SSD as was ext4. These recommendations came from a blog by Theodore Ts'o, the lead maintainer of ext4.

Two benchmarks were used for testing - IOzone, which was used for throughput and IOPS testing, and metarates for metadata testing. A total of 13 throughput tests were performed, 4 IOPS tests, and 3 metadata tests. Each particular test was run 10 times and the average and standard deviation were presented in this article.

In general the tests showed little difference between the "before" performance and the "after" performance. However, there were some differences:

  • Random Read Throughput: The "before" is much better at record sizes of 4MB, 8MB, and 16MB, with the differences being about 13-15% in favor of the "before" results.
  • Random Write Throughput: At a record size of 4MB, the performance before the stress test ("before") was about 20% better than "after" the stress test
  • Backwards Read Throughput: The "before" performance is better by about 17-20% for the 4MB, 8MB, 16MB record sizes.
  • Strided Read Throughput: The "before" performance is better by about 22-23% for the 4MB, 8MB, 16MB record sizes
  • freread Throughput: The "before" performance is better by about 9% to 13% for the 4MB, 8MB, and 16MB record sizes.
  • Random Write IOPS: The random write IOPS performance at a record size of 32KB was better by about 10.82% before the stress testing than after. At a record size of 64KB the performance is better by about 8.73%
  • Random Read IOPS: The "after" performance is better at a 32KB record size (14.26% better) and at a 64KB record size (12.6% better).
  • File close/open metadata: At NP=4, the "before" performance is is about 9% better than the "after" stress testing performance
  • File stat: There is some difference in performance between the before and after testing with the "after" performance being 3.88% better at NP=1, 6.58% at NP=2, and 3.44% at NP=4.

 

From these results, it looks like the Intel X25-E performed well even after being severely stressed. There are comparisons where the performance before the stress testing was noticeably better than after the stress testing, but there were also benchmarks were the reverse was true (i.e. the performance after the stress test was better). But I think the differences were reasonably small enough over a range of I/O patterns (except for a few unusual ones) that it is safe to say that this particular enterprise class SSD has effectively used various technologies to avoid performance degradation over time.

Sweeping generalizations are difficult to avoid after doing so much testing, but the results indicate that this particular enterprise class SSD has performance that only varies slightly after the drive has been subjected to some fairly severe IO stress. Moreover, it is almost impossible to link cause and effect, but given the poor performance of the early SSDs, it is fairly safe to conclude that the performance and longevity techniques mentioned in this article series, and possibly others, have contributed to this drive being able to perform well over time.

Back to Page 1

Jeff Layton is the Enterprise Technologist for HPC at Dell, Inc., and a regular writer of all things HPC and storage.

Follow Enterprise Storage Forum on Twitter.


Page 4 of 4

Previous Page
1 2 3 4
 

Comment and Contribute

 


(Maximum characters: 1200). You have characters left.

 

 

Storage Daily
Don't miss an article. Subscribe to our newsletter below.

Thanks for your registration, follow us on our social networks to keep up-to-date