Benchmarking has long been a contentious issue among storage vendors, but a recent Network Appliance benchmark has raised the controversy to a new level.
It all started last year when Network Appliance decided to benchmark its FAS3040 modular storage system and EMC’s Clariion CX3 Model 40 under the auspices of the Storage Performance Council’s benchmark program. In the past, vendors have benchmarked just their own products. In this case, NetApp chose to compare its offering to one from EMC, a company that has studiously avoided benchmarking its products (see Hitachi Reopens Benchmark Debate).
The unusual move to include a competitor’s product didn’t grab headlines because the council is under no legal obligation to notify vendors of such activity or announce such council member decisions, said SPC administrator Walter Baker.
The council’s lone requirement is that vendors not skirt legal stipulations that may preclude a product from being benchmarked without vendor authorization, Baker said. According to the SPC, NetApp followed the rules and requirements in conducting the testing and completing the SPC’s auditing mandates.
While EMC isn’t saying NetApp broke any laws, it clearly believes it has stepped way out of bounds in publishing its evaluation results.
EMC Global Marketing CTO Chuck Hollis describes the SPC benchmark program and NetApp’s activity as “bench marketing.”
“Benchmarking, true objective testing, is very important to us. We spend a stupid amount of money on research and testing. What this is [the NetApp test results] is free entertainment based on cheap marketing tactics,” said Hollis.
NetApp ran the two SAN products through SPC’s SPC-1 benchmark with and without snapshot technology enabled. Snapshots are used by enterprises to provide continuous protection and smart copies of data.
According to the published results, NetApp beat the Clariion in performance. According to SPC guidelines, EMC has 60 days to challenge or protest the benchmark results. As of last week, the SPC said it had not received any challenge notification.
The benchmark results aren’t impressing most industry analysts for several reasons. Some cited the fact that an objective party did not oversee the testing. Another expert said a one-time evaluation effort typically holds little validity.
Charles King, principal analyst at Pund-IT, said it’s pretty much a matter of “he said, she said,” since vendor-sponsored tests bring lots of “wiggle room” into play.
“In a one-time case, it’s more of a vendor rubbing another vendor’s face in the dirt. But if this isn’t a one-time deal it could present an interesting situation,” said King.
And that’s where the showdown looms. NetApp plans to keep testing its products against EMC using the SPC benchmark program. “We will certainly do it again. This was not done in isolation,” said Val Bercovici, NetApp’s chief technology architect.
“EMC was ratcheting up erroneous performance information about our products and making claims about its own products that we knew were not correct. They called our performance into question and so we decided to put the products to the test to show customers which performs best,” Bercovici said.
“We proved that our product doesn’t take any performance hits when snapshot is in play, while it obviously impacts the EMC product performance,” he said. “The competition forced this benchmark. We just want to put the right information out there to customers.”
— Charles King
Hollis, for his part, is also concerned about getting customers correct information. He’s worried enterprises not familiar with storage technologies may not be able to decipher between true benchmarks and a marketing gimmick.
“We’re concerned about the customers who might not know better,” Hollis said. “There’s an ethical line here and EMC will not cross it. What we’re telling them [new customers] is we’ll run a benchmark test for you that relates to your actual environment to show you what the real performance factor is with our products.”
King, as well as Illuminata analyst John Webster, are concerned with the SPC’s role in the benchmarking activity and what the council’s intentions are in allowing such competitive testing.
“My issue is whether or not we can regard it as unbiased and independent,” Webster wrote in a recent column. “In spite of the fact that we have here the unprecedented situation where vendor X does a benchmark run on a competing vendor’s product, the SPC made no public statement with regard to the operative policy under which this benchmark was published.”
EMC is a client of Illuminata but the vendor said it had nothing to do with Webster’s commentary.
In response, Baker said the benchmarking process is open and transparent. “They clearly are not understanding what we’re trying to do,” he said. “There is no downside to being completely open, and that’s what we’ve been.”
King wondered why the SPC would allow itself to be “dragged” into such a unique situation. “The SPC may have just opened the doors to a new era that could change the landscape of benchmarking as we know it,” he said.
Article courtesy of InternetNews.com