The Art of Storage Benchmarking Page 3
Benchmarking Evaluation Process
The ideal way to ensure that the process is fair is by attending the final benchmark results to witness the running of your operational environment, certify the results, and ensure that the vendor does not try anything you missed in the rules. Most of the time, however, visiting each vendor is impossible given the costs and the time. Most often, what happens is that you provide the instructions to the vendor and wait for the vendor presentation with the results, and then decide who the winner is based on the decision criteria.
Ah, what about the criteria? In my experience, this is one of the most important parts of the process. The benchmark team must agree as to the evaluation criteria before the benchmark is released. Everyone has their own favorite vendor from accounting to the system manager to the operations manager to you. Having agreed upon written evaluation criteria is the only way to be fair to evaluate the vendors and eliminate in-fighting within the organization after the results are delivered. I am not suggesting in any way that the vendors be given the evaluation criteria, but I am strongly recommending that you have internal agreement on the evaluation criteria before the benchmark is released.
OK, what should the criteria for evaluation be? Back in the mid 1990s, there was a saying about RAID purchases. You could have it Fast, you could have it Reliable, you could have it Cheap -- pick any two. Though RAID has evolved quite a bit, the same criteria still applies for the most part to the evaluation of any system or set of hardware components -- the evaluation criteria is a balancing act unless you have an unlimited budget, and very few do.
In other words, you cannot get 99.999% (also known as 5 9s) up time (about 2 minutes of down time per year) with equipment that costs the same as equipment that will only give you 99.9% up time (about 8 hour and 45 minutes of down time per year) and still have the same performance. You are not going to buy the equipment/system to support 5 9s of up time for even two times the prices of three nines and get the same performance. Therefore, trading off performance, reliability, and price remains quite similar to that old saying about RAID (if a saying in our industry that was used 10 years ago is old).
Developing an acceptance plan should be part of the benchmarking process. Vendors should be informed of the criteria for acceptance before they submit the final proposals, which are often called Best and Final Offerings (or BAFO). The acceptance plan should therefore be agreed upon by internal interested parties. The acceptance plan should integrate the hardware and/or software into your operational environment, and provide the performance and reliability that the environment requires. More often than not, the hardware arrives on the dock and is powered up, and the accounting department cuts a check. I do not feel this is good, either for the vendor or for the customer.
For the vendor acceptance criteria, ensure that the equipment is delivered, configured to meet the benchmark criteria, and works.
For the customer acceptance criteria, ensure that you get to see the benchmark performance, how to configure the system, the configuration options, and therefore some free training.
Having an acceptance process provides mutual benefits for both parties and should be part of the procurement process. In almost all cases I have seen, when acceptance criteria are clearly defined and an acceptance process is required, the new hardware is more quickly integrated into the environment. This reduces the real cost for the hardware and/or software, and also reduces the possibility of in-fighting within the procurement group.
Having a formal process for the acquisition of new hardware provides a great deal of benefits for both the vendors and the customer. Of course, you are often just buying a single RAID controller or single HBA to increase what you already have, but that does not preclude you from thinking about newer models or other vendors if the price and performance are a big difference. With the speed of change when it comes to technology, buying older stuff just because it is the same given that most parts are interoperable often does not make sense.