This month we will cover benchmarking systems. Now that we have covered the data path in full, I think it is important to follow with a discussion of benchmarking.
Understanding the benchmarking process is important even if you are not going to require a benchmark. We're going to cover the benchmarking process as an introduction to the storage architecture and storage decision process. My goal is to provide:
- An understanding of the ins and outs of the benchmarking process
- Some insight into the inner workings of the vendors during this process
- Some suggestions on how to make the process fair for both parties (the vendor and the purchaser)
Defining a good benchmark that is fair to the vendors and more importantly fair to the purchaser is very difficult.
What Is Fair
Very few vendors generally want to conduct a benchmark, and storage vendors shy away from benchmarking even more than most. The cost of benchmarking storage is very expensive. You not only have the costs of the server(s), HBAs, fibre channel switch(s), and RAID(s), but you're also likely to have the software expense of the file system and/or volume manager. Couple that with the fact that you need a good applications analyst to understand the benchmark I/O access patterns, a system tuner, a RAID guru, someone to write the report, and a project manager.
If you require a benchmark, you will force the vendor to incur a huge cost in running a proper benchmark. On the other hand, if you are going to be purchasing over $500K of storage, you really need to see if any given vendor with their product set can meet your requirements. You cannot expect a vendor to run a benchmark that costs them $40K for a system to be purchased that is less than $500K.
A good rule of thumb is that the cost of the benchmark should be less than 2% of the total system value. You might be able to get that to 5%, but keep in mind that the vendors will recover the cost of the benchmark one way or another if they conduct a successful benchmark. In many cases, for purchases under $500K you would likely be better off hiring a storage expert rather than running a benchmark.
Assuming that your purchase amount is significant enough to warrant a benchmark, you'll need to follow some basic steps.
The analysis of your requirements is key to developing a benchmark that meets your organization's needs. If you are benchmarking storage, you need to look at your requirements. These include:
- Backup recovery
- Reliability and repair time
- Application usage
- File system being used
- HBA Failover
Benchmarking storage is difficult because there are so many levels of indirection. You have to consider a number of questions, including:
- What will the size of the application I/O request be?
- What does the system do with that request?
- How and what does the file system and/or volume manager do with the request?
- What happens in the HBA with the queue of requests?
- If a switch is involved, how will issues with latency performance be understood and characterized?
- How will the RAID controller command queue manage the I/O requests?
- How will the RAID controller cache and caching algorithm work?
- What will be the I/O performance for the controller and the I/O performance of the disk drives and the disk cache?
- What happens when an HBA, switch, and/or RAID fails?
Understanding all of this is not a straightforward process at best; therefore, to do it correctly, the vendor will incur a high labor cost to run a benchmark. You need to carefully review your requirements before you set up a benchmark and use these requirements to develop a detailed but reasonable benchmark specification.
Development of benchmarking rules is just plain hard work. Remember, the goal of every benchmarker is to win every benchmark. That is often how the person earns a good portion of their salary. Therefore, a good benchmarker reads your rules and looks for advantages and loopholes that can often be seen only by experienced benchmarkers.
As a benchmarker, I often read the rules and followed the rules to the letter, not necessarily the spirit. In the past (I am a reformed benchmarker), I was known for developing SBTs (slimy benchmarking tricks) -- things like creating file systems with allocation equal to the file sizes in the benchmark to optimize the allocation. Other things I did were equally shady, such as placement of the file system in memory, and booting from an SSD to speed UNIX pipes for an interactive benchmark. In each of these cases, the rules were followed to the letter, but the customer did not get what they wanted, although they did get what they asked for. My company was happy when we won the benchmarks, and I received a bonus for every win.
What should be clear from all of these examples is that documenting what you really want and how you want it done is critical. As a reformed benchmarker, I now spend some time writing benchmarking rules. For a recent benchmark for a government organization, a detailed set of rules for running the benchmarks totaled 100 pages. Obviously, this is a very time consuming practice, and is not very cost effective unless you have a very large procurement or you have regular large procurements. The government organization in question has a yearly benchmark process to purchase around $40 million in new hardware.
Much of the benchmarking that you might be involved with will include databases, which presents numerous problems when designing benchmarks and the rules. How and where the tables, indexes, redo logs, and even the database application itself reside become critical issues. The absolute most important part of any benchmark is to test your reality. Does the benchmark truly match your current environment or the environment you expect to be operating in? Operational issues like backup and recovery must fit within the design of a benchmark as well. For example, if you are purchasing storage, you must have the vendor benchmark the database on the server you use. The same holds true with server benchmarks for storage.
Benchmarking Evaluation Process
The ideal way to ensure that the process is fair is by attending the final benchmark results to witness the running of your operational environment, certify the results, and ensure that the vendor does not try anything you missed in the rules. Most of the time, however, visiting each vendor is impossible given the costs and the time. Most often, what happens is that you provide the instructions to the vendor and wait for the vendor presentation with the results, and then decide who the winner is based on the decision criteria.
Ah, what about the criteria? In my experience, this is one of the most important parts of the process. The benchmark team must agree as to the evaluation criteria before the benchmark is released. Everyone has their own favorite vendor from accounting to the system manager to the operations manager to you. Having agreed upon written evaluation criteria is the only way to be fair to evaluate the vendors and eliminate in-fighting within the organization after the results are delivered. I am not suggesting in any way that the vendors be given the evaluation criteria, but I am strongly recommending that you have internal agreement on the evaluation criteria before the benchmark is released.
OK, what should the criteria for evaluation be? Back in the mid 1990s, there was a saying about RAID purchases. You could have it Fast, you could have it Reliable, you could have it Cheap -- pick any two. Though RAID has evolved quite a bit, the same criteria still applies for the most part to the evaluation of any system or set of hardware components -- the evaluation criteria is a balancing act unless you have an unlimited budget, and very few do.
In other words, you cannot get 99.999% (also known as 5 9s) up time (about 2 minutes of down time per year) with equipment that costs the same as equipment that will only give you 99.9% up time (about 8 hour and 45 minutes of down time per year) and still have the same performance. You are not going to buy the equipment/system to support 5 9s of up time for even two times the prices of three nines and get the same performance. Therefore, trading off performance, reliability, and price remains quite similar to that old saying about RAID (if a saying in our industry that was used 10 years ago is old).
Developing an acceptance plan should be part of the benchmarking process. Vendors should be informed of the criteria for acceptance before they submit the final proposals, which are often called Best and Final Offerings (or BAFO). The acceptance plan should therefore be agreed upon by internal interested parties. The acceptance plan should integrate the hardware and/or software into your operational environment, and provide the performance and reliability that the environment requires. More often than not, the hardware arrives on the dock and is powered up, and the accounting department cuts a check. I do not feel this is good, either for the vendor or for the customer.
For the vendor acceptance criteria, ensure that the equipment is delivered, configured to meet the benchmark criteria, and works.
For the customer acceptance criteria, ensure that you get to see the benchmark performance, how to configure the system, the configuration options, and therefore some free training.
Having an acceptance process provides mutual benefits for both parties and should be part of the procurement process. In almost all cases I have seen, when acceptance criteria are clearly defined and an acceptance process is required, the new hardware is more quickly integrated into the environment. This reduces the real cost for the hardware and/or software, and also reduces the possibility of in-fighting within the procurement group.
Having a formal process for the acquisition of new hardware provides a great deal of benefits for both the vendors and the customer. Of course, you are often just buying a single RAID controller or single HBA to increase what you already have, but that does not preclude you from thinking about newer models or other vendors if the price and performance are a big difference. With the speed of change when it comes to technology, buying older stuff just because it is the same given that most parts are interoperable often does not make sense.