Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
In the storage industry, looks can be deceiving.
Established data storage vendors seem to have it pretty good: margins are healthy, barriers to entry appear high, and new technology startups tend to get acquired before they pose a threat to the status quo. The top half a dozen or so vendors haven't been challenged for quite some time.
Yet the data storage industry will likely be turned on its head over the coming years: storage in 2024 will likely look very different from how it looks today, and there are a number of reasons why.
Data Storage Future and the Cloud
The first reason comes down to the cloud, and public clouds offered by the likes of Amazon, Google and Microsoft. The potential benefits of public clouds are well known and there is no need to repeat them here, but suffice it to say that when you move workloads to the cloud you pay only for the resources you need. The instant scalability the cloud provides makes organizations highly agile.
That being the case, companies are increasingly moving workloads to the cloud, and that trend looks likely to continue: investment bank Piper Jaffray predicts a 44% annual growth in workloads running in the public cloud for the next five years.
And here's the rub. "As you move workloads to the cloud, you need less storage, so that is bound to have an impact on the storage industry," says Arun Chandrasekaran, an analyst at Gartner.
Of course, most businesses won't move their ERP systems or their key database applications to the cloud just yet – but certainly backup and other tier 2 and 3 applications will likely be moving that way in the near future.
Clearly those workloads that move to the cloud will still need data storage to go with them, but the bad news for storage vendors is that storage won't come from them. "Hyper-scale cloud providers are not buying their storage from established storage vendors - they build their own," Chandrasekaran points out.
Large scale Internet companies like Facebook, and initiatives like the Open Compute Project are driving the development of large cloud-scale storage systems that companies can assemble themselves from cheap building blocks including commodity disk drives. Each of these replaces large numbers of traditional storage vendors' offerings.
Data Storage Future: Open Source, Flash, SDS
Another reason that the storage landscape is changing is the rise of open source storage solutions. "These are not new, but what's changed is that there are foundations and frameworks like OpenStack and Hadoop that are now ready," says Chandrasekaran.
He points out that many of these foundations are based on commodity servers or direct attached storage (DAS) - Hadoop is built mostly on DAS, for example.
This type of massively scalable open source storage software is increasingly being used in production environments, and when it is, it is inevitably replacing traditional storage vendors' offerings.
A third source of disruption is flash-based storage. The situation here is a little different from the previous two in that many traditional vendors are embracing flash technology, and as yet it has not seen large scale penetration in large scale environments such as those operated by cloud providers.
But it is disruptive, and that's because it decouples performance from the number of spindles being used. Systems that use flash can provide very high IOPS as an alternative to running vast numbers of short stroked disks, which traditional vendors are only too happy to supply in storage systems at high margins.
Having said that, it is often traditional storage companies that are supplying the flash functionality, as Chandrasekaran points out. "A lot of flash is deployed by established vendors, and that's because they are taking stock and taking steps to stay relevant."
The fourth reason comes down to software defined storage (SDS), and the fact that software – when combined with commodity hardware – is powerful enough that you don’t need intelligence at the hardware layer any more. Chipmakers like Intel have made innovations in the x86 architecture which allow their processors to do the heavy lifting that's required.
"This is enabling a new breed of start up firms who can focus on the software layer and come to market quickly," says Chandrasekaran. He points out that creating new products is much easier for this group. "The truth is that "new" storage vendors like Red Hat or VMware (with Virtual SAN, for example) have no legacy products, no innovator's dilemma like established vendors have, no incumbency problems. So software is coming to the forefront."