Benchmarking Storage Systems, Part 3 Page 2
So how do you score all of this? Here are some scoring examples from some real benchmarks I have seen and participated in:
- Price alone – We need this much RAID-X storage, with the lowest price winning
- Price and performance and reliability requirements – We need this much RAID-X storage, you must meet this performance criteria and these requirements, and the reliability must meet these requirements
- Price, reliability and performance scaling – We need this much RAID-X storage and you must meet a minimum requirement, but vendors are encouraged to submit bids with better than the minimum performance. The reliability must meet these requirements
- Performance is the dominating factor – A few bids require performance near the edge of what is currently commercially available. For these bids, price is a consideration but performance dominates, as the customer must be willing to pay for this N-case requirement. Reliability is also defined in this type of procurement
The Real Price
One additional area to consider is the real price. The real cost of storage hardware is not just the purchase price and maintenance price, but also includes the cost to operate the system. This cost includes:
- Training your personnel
- In some cases, the cost of maintaining spare parts
- In some large sites, the cost of having vendor personnel on-site and the space that is required
All of these must be considered in the actual pricing model section of the scoring model that you need to develop.
As you can see, the different types of scoring run the gambit from price centric to performance centric. This is why it is so critical for an organization doing a procurement to all agree on the scoring model before the procurement goes out. If this is not done, the procurement often drags on while the various organizations fight the political battles of who has the most important requirements.
One thing that should go without saying but that is sometimes is lost in the fray is the importance of not allowing the vendors to get their hands on the scoring models. If they do get the models – even without the pricing models – you will end up with everyone providing the same price/performance ratios. In over 23 years of being involved with benchmarks, I have never seen a case where it is in the buyer’s best interest to provide the scoring model. When this does happen vendors tend, and rightly so, to try to optimize the scoring model with their architecture rather than providing the best bid with the hardware they can propose.