The Art of Storage Benchmarking

Benchmarking Systems

This month we will cover benchmarking systems. Now that we have covered the data path in full, I think it is important to follow with a discussion of benchmarking.

Understanding the benchmarking process is important even if you are not going to require a benchmark. We're going to cover the benchmarking process as an introduction to the storage architecture and storage decision process. My goal is to provide:

1. An understanding of the ins and outs of the benchmarking process
2. Some insight into the inner workings of the vendors during this process
3. Some suggestions on how to make the process fair for both parties (the vendor and the purchaser)

Defining a good benchmark that is fair to the vendors and more importantly fair to the purchaser is very difficult.

What Is Fair

Very few vendors generally want to conduct a benchmark, and storage vendors shy away from benchmarking even more than most. The cost of benchmarking storage is very expensive. You not only have the costs of the server(s), HBAs, fibre channel switch(s), and RAID(s), but you're also likely to have the software expense of the file system and/or volume manager. Couple that with the fact that you need a good applications analyst to understand the benchmark I/O access patterns, a system tuner, a RAID guru, someone to write the report, and a project manager.

If you require a benchmark, you will force the vendor to incur a huge cost in running a proper benchmark. On the other hand, if you are going to be purchasing over $500K of storage, you really need to see if any given vendor with their product set can meet your requirements. You cannot expect a vendor to run a benchmark that costs them $40K for a system to be purchased that is less than $500K.

A good rule of thumb is that the cost of the benchmark should be less than 2% of the total system value. You might be able to get that to 5%, but keep in mind that the vendors will recover the cost of the benchmark one way or another if they conduct a successful benchmark. In many cases, for purchases under $500K you would likely be better off hiring a storage expert rather than running a benchmark.

Page 2: Requirements

0 Comments (click to add your comment)

Comment and Contribute

 

Your name/nickname

Your email

Subject

(Maximum characters: 1200). You have characters left.