Benchmarking Storage Systems, Part 3 Page 4
Description of Overall Environment and Environmentals
You should provide a detailed description of the site(s) and the environmentals. This should include:
- Information on available floor space
- Information on power and cooling
- Current floor loading limitations
- Shipping address(es)
As Clint Eastwood said in Magnum Force: “A man’s got to know his limitations,” and vendors need to understand the environment and facilities they’ll be working with.
The cost proposal should detail all of the requirements for pricing the system, including:
- Hardware costs
- Software costs
- Hardware and software maintenance and upgrade policy
- Professional services
This section often provides instructions that define the requirements for services and asks for a single price to simplify the information from each vendor.
Benchmark Description and the Rules
This is by far the most difficult section to write. The complexity increases with the number of benchmark applications that you run. The greater the number of applications the more complexity you will have in running the applications and checking the results.
You need to write the rules in such a way that the vendors clearly understand what is expected of them. The rules should be written such that every vendor is running your benchmarks in exactly the same way.
You want to benchmark the systems, not benchmark the benchmarkers. For example, when benchmarking a RAID you must specify the number of HBAs and from what hosts the RAID will be attached to. Without doing this you might find that a vendor uses more HBAs and attaches to a host with a faster PCI or PCI-X bus. Leave nothing to chance — specify as much detail as possible.
This series on benchmarking provides a good overview of the benchmark process from both the vendor’s and the purchaser’s points of view. As I’ve illustrated, there are many obvious and not-so-obvious aspects to the benchmark process that need to be considered, from the politics inside the purchaser’s organization to the technical issues of how to develop the benchmark.
On the vendor side of the fence, they are trying to figure out what you really want and how much you can pay for it, and most importantly which combination of hardware and software products will meet the requirements for the benchmark at the best cost. Notice I say meet the requirements and not necessarily win the benchmark, because this becomes the price/performance tradeoff. Procurements generally come down to cost and performance, and from the vendor side, they are trying to determine what you can afford to pay for the hardware, software, and services you have specified.
It’s a game that is played every day, with the purchaser trying to get the best price that meets the requirements, and the vendor trying to win the business with the highest margins. It’s also an extremely complex game, so if you’re new to the game, it’s a good idea to find someone to help you with rules.