Benchmarking Storage Systems, Part 1 Page 2
Component Part Benchmarking Issues
Each component can have a significant impact on the results of your benchmark. Let's take a quick look at each.
RAID and Tunables
Benchmarking RAID devices can be difficult given all of the different aspects of the RAID hardware and software:
- Front-end performance – Performance from the host to the Fibre Channel connection to the cache
- Cache – Performance and bandwidth of the cache and the caching algorithm
- Back-end performance – Performance of the RAID from the cache to the disk
Add to these issues the performance of various RAID levels, the myriad of tunable parameters for cache, cache allocation, device allocation, etc., and you will soon realize that it’s extremely difficult to cover every area of RAID hardware and software.
Instead, you need to ensure that you understand the effects of the I/O of your benchmark and how the real workload utilizes the cache of the RAID in a way that will emulate the behavior of the real system.
Tape and Libraries
Benchmark tapes and libraries have the fewest number of hardware and software areas that need to be considered. Tape compression is a big consideration for performance, along with the interface (see this article for more on the issue).
The tricky part is the development of data sets that closely mimic your data to ensure that the compression for the benchmark matches the compression at purchase. The other area to consider is the tape load and position time. For libraries, the issues are the robot pick time and how well the library will work with the software that you’re going to be using.
Benchmarking file systems is just plain hard and is likely to be the most complex part of the benchmark. It also has the potential to create quite the opportunity for a large number of SBTs (I should know, as I used to do them, but have since reformed). Here are a few areas to consider:
- Allocation sizes
- Topology of the file system metadata, data, and log, and RAID devices
- Tunables for allocation, logs, metadata, etc.
- Server memory and file system tunables
By far the biggest issue I have seen is the discrepancy in performance for file systems when they are fragmented and when they are just mkfs’s. More than likely, your storage benchmark would run on a newly mkfs’ed file system, which would likely lead to larger I/O requests and could change the results of the benchmark for the RAID vendor, as some vendors are far better at large I/O requests compared to smaller requests, and will surely change the RAID tunable’s large block performance.
It cannot be emphasized enough that you need to either specify the system tunables for the benchmark or ensure that each vendor reports the tunable changes.