In the storage industry, looks can be deceiving.
Established data storage vendors seem to have it pretty good: margins are healthy, barriers to entry appear high, and new technology startups tend to get acquired before they pose a threat to the status quo. The top half a dozen or so vendors haven't been challenged for quite some time.
Yet the data storage industry will likely be turned on its head over the coming years: storage in 2024 will likely look very different from how it looks today, and there are a number of reasons why.
Data Storage Future and the Cloud
The first reason comes down to the cloud, and public clouds offered by the likes of Amazon, Google and Microsoft. The potential benefits of public clouds are well known and there is no need to repeat them here, but suffice it to say that when you move workloads to the cloud you pay only for the resources you need. The instant scalability the cloud provides makes organizations highly agile.
That being the case, companies are increasingly moving workloads to the cloud, and that trend looks likely to continue: investment bank Piper Jaffray predicts a 44% annual growth in workloads running in the public cloud for the next five years.
And here's the rub. "As you move workloads to the cloud, you need less storage, so that is bound to have an impact on the storage industry," says Arun Chandrasekaran, an analyst at Gartner.
Of course, most businesses won't move their ERP systems or their key database applications to the cloud just yet – but certainly backup and other tier 2 and 3 applications will likely be moving that way in the near future.
Clearly those workloads that move to the cloud will still need data storage to go with them, but the bad news for storage vendors is that storage won't come from them. "Hyper-scale cloud providers are not buying their storage from established storage vendors - they build their own," Chandrasekaran points out.
Large scale Internet companies like Facebook, and initiatives like the Open Compute Project are driving the development of large cloud-scale storage systems that companies can assemble themselves from cheap building blocks including commodity disk drives. Each of these replaces large numbers of traditional storage vendors' offerings.
Data Storage Future: Open Source, Flash, SDS
Another reason that the storage landscape is changing is the rise of open source storage solutions. "These are not new, but what's changed is that there are foundations and frameworks like OpenStack and Hadoop that are now ready," says Chandrasekaran.
He points out that many of these foundations are based on commodity servers or direct attached storage (DAS) - Hadoop is built mostly on DAS, for example.
This type of massively scalable open source storage software is increasingly being used in production environments, and when it is, it is inevitably replacing traditional storage vendors' offerings.
A third source of disruption is flash-based storage. The situation here is a little different from the previous two in that many traditional vendors are embracing flash technology, and as yet it has not seen large scale penetration in large scale environments such as those operated by cloud providers.
But it is disruptive, and that's because it decouples performance from the number of spindles being used. Systems that use flash can provide very high IOPS as an alternative to running vast numbers of short stroked disks, which traditional vendors are only too happy to supply in storage systems at high margins.
Having said that, it is often traditional storage companies that are supplying the flash functionality, as Chandrasekaran points out. "A lot of flash is deployed by established vendors, and that's because they are taking stock and taking steps to stay relevant."
The fourth reason comes down to software defined storage (SDS), and the fact that software – when combined with commodity hardware – is powerful enough that you don’t need intelligence at the hardware layer any more. Chipmakers like Intel have made innovations in the x86 architecture which allow their processors to do the heavy lifting that's required.
"This is enabling a new breed of start up firms who can focus on the software layer and come to market quickly," says Chandrasekaran. He points out that creating new products is much easier for this group. "The truth is that "new" storage vendors like Red Hat or VMware (with Virtual SAN, for example) have no legacy products, no innovator's dilemma like established vendors have, no incumbency problems. So software is coming to the forefront."
Software defined storage brings other headaches to established vendors beside the fact that it opens the door to storage start-ups and companies like Red Hat and VMware who are already well recognized outside the storage arena.
One of these is that it can lead to "performance convergence." If all storage systems are based on commodity hardware – including Intel Xeon multicore processors rather than painstakingly developed proprietary ASICs – and all are introducing SSDs into the mix, then it gets harder to produce a product that offers storage performance that is significantly better than competitors. That means it is harder for vendors to justify selling systems at a premium price unless they can offer innovative and useful storage features that other vendors don't.
Future Impacted by ‘Compressed Differentiation’
But here's the problem: vendors are increasingly suffering from what Chandrasekaran terms "compressed differentiation." What he means by this is that previously vendors could introduce new features into their storage systems and enjoy a lead over rivals of a couple of years before they caught up, the increasing importance of software means that vendors can only hope for a three or four months lead before competitors catch up and introduce similar – or even superior – features.
That being the case, there is little incentive for storage users to switch vendors to get "killer" storage features. They may as well just sit it out for a few months for their current vendor to catch up.
So what does all this mean for established storage vendors? "The traditional SAN has a finite future," says Mark Peters, an analyst at Enterprise Strategy Group. "How long will it be before most data is stored centrally - maybe in the cloud? We will see that in under a decade."
At a Gartner IT Operations and Datacenter Summit held in Sydney in May, Chandrasekaran outlined a number of possible outcomes for the longer term future of the storage industry including:
· Market share leaders losing the most
· Large portfolio vendors giving up
· Smaller companies getting crushed
· Smaller companies becoming more agile, and being perceived as delivering more value
While it's too early to tell which of these scenarios will play out, he is a little more optimistic about the future of established vendors in the short term. "I think that SANs will still be around in five years' time for tier 1 workloads, and many customers will continue to invest in them," he says. "But the vendors will struggle to get more than single digit growth and in some segments it will be close to zero."
When it comes to non-mission-critical workloads, the outlook is bleaker for established vendors. Storage professionals are conservative so any shifts won't happen quickly, but as they gain confidence in cloud-based solutions and other open source systems, the shift away from big ticket proprietary systems and towards open source and cloud solutions will accelerate.
The good news for vendors is that there are barriers to exit from their ecosystems, and there are reasons that customers may stick with them. "Big vendors have relationships with key enterprise ISVs and a large proportion of storage purchases are done by workloads," Chandrasekaran points out. "So vendors will try to maintain relevance with workload-level differentiation."
There's also a good reason for systems vendors who are involved in storage to stick with it. "Systems vendors can't think of storage in isolation," he says. "They need to compete (in the storage space) because if they don't have a storage offering they have no converged story or strategy."
But as margins are impacted – which they surely will be – some big vendors may have to decide whether there is a long term future in their business at all.
Photo courtesy of Shutterstock.