Many of us remember those old computer messages that no more storage space was available on the hard drive. The user had no choice but to offload some data or spend some hours going through the files finding material that could be deleted. Enterprise users will also recall the fondness some storage administrators had for sending out those “You have exceeded your storage quota” notices. Fortunately, the size of today’s hard disk drives (HDDs) seems to have brought about a virtual disappearance of such unwelcome communications.
But what about the storage world as a whole? We are merrily merely packing information onto digital storage from every conceivable angle. Mobile phones, big data, the cloud, tablets, the internet of things (IoT), analytics and more are gobbling up available storage capacity at an alarming rate.
Could we reach the point where there is simply nowhere left to put all this data? In other words, could we arrive at Storage Armageddon or Stor-mageddon?
Over the next few weeks in this series of articles, Enterprise Storage Forum will investigate the known and unknown digital universe in terms of its current size as well as future capacity requirements. In addition, we will look at developments in technologies such as tape, disk, flash and storage management to see what solutions could help us avoid a potential storage end of days.
The Digital Universe
The size of the digital universe is staggering. People have gotten so used to bandying around storage capacity acronyms that they may have lost sight of what they really mean. So let’s begin by covering some basics. Terabyte (TB) drives have become quite common – a TB is a trillion bytes. Now multiply a TB by a thousand and you have a petabyte (PB), another thousand is an exabyte (EB) and another 1000 gives you a zettabyte (ZB). According to International Data Corp.( IDC), 4.4 ZB of digital data had been created by 2013. That number is predicted to reach 44 ZB by 2020.
IDC’s Digital Universe report provides many interesting tidbits on this problem. According to Vernon Turner, an analyst at IDC, data growth is moving at a rapid clip – about 40 percent per year.
“The world’s available storage capacity is growing slower than the digital universe,” said Turner.
At the start of last year, IDC estimated that available storage capacity could hold 33 percent of the total. Within five years, the sheer volume of data created means that less than 15 percent can be stored. There are many facets to this. Certainly, the fact that much of this data is transient (streamed video, phone and text traffic, etc.) serves to ease the pressure somewhat.
But even that transient data still gobbles up an awful lot of capacity (at least temporarily). Then there is the metadata aspect and the data transmission side. Just how much storage and bandwidth get used up by such viral hits as an Ellen DeGeneres/Bradley Cooper celeb selfie at the Oscars (26 million views in 12 hours)?
Cisco’s Global Mobile Data Traffic Forecast presents a frightening prospect. Let’s take up just a handful of its data points: Mobile data grew 69 percent last year, reaching 2 EB per month. The total worldwide mobile traffic for 2014 alone exceeded the entire extent of the internet in 2000. Mobile video traffic now accounts for more than half the overall volume. With about 7.5 billion mobile devices in operation and many more of these being converted to smartphones (a smartphone generates 22 times more traffic than a non-smart device), the mobile traffic burden is mushrooming. Factor in the fact that connection rates are growing at around 20 percent per year. As 4G service expands, it should be noted that 4G devices churn through 10 times more data than those of earlier generations.
What about tablets? There were 74 million mobile-connected tablets at the end of 2014, a 1.6 fold increase over the previous year, and a tablet consumes 2.5 times more traffic than a smartphone. And how about the latest wearable craze? Who knows how much data that will ultimately add to the mix?
Cisco predicts that monthly global mobile traffic will surpass 24.3 EB by 2019, that smartphones will reach 75 percent of mobile data traffic by 2019, that monthly mobile tablet traffic will exceed 2 EB by 2018, and that tablets will account for 10 percent of total mobile traffic by 2017. Further, it says three quarters of the world’s mobile data traffic will be video by 2019.
“As more and more businesses capitalize on the social and mobile phenomenon, the enormity of the digital universe grows,” said Jeremy Burton, president, products and marketing, EMC. “While the potential is massive, the implications are daunting.”
To drive home the point some more and boggle a few more minds, let’s bring IoT into the equation. IDC currently estimates that 14 billion IoT devices and sensors connected to the internet globally eat up only 2 percent of the digital world. Yet there are 200 billion potential IoT connection points already in existence.
“By 2020, the number of IoT connected devices will grow to 32 billion, representing 10 percent of the digital universe,” said Turner.
That may well turn out to be an underestimate. Some are already talking about a trillion sensor networks within 10 years.
Clearly, this represents a massive challenge for storage vendors. It’s something like a jet fighter challenging a Toyota Corolla to a race across the country. With each passing mile, the jet gets further and further away. Storage innovation, then, is going to have to come up with new ways to store more data more densely and transmit data more rapidly. Fortunately, developments are taking place on many fronts as no one technology or breakthrough is going to resolve this challenge.
Some that promise to at least alleviate the problem are already hitting the market. The next few years will likely see major breakthroughs in HDD, flash-based solid state drives (SSD) and tape which will greatly increase capacity, density and throughput. All of this is required to face the growing storage shortfall.
“It is going to mean using HDDs, flash and tape in many different ways if storage is going to keep up,” said Greg Schulz, an analyst with StorageIO Group. “We will also need far more efficient storage management and software-defined architectures.”
Over the coming weeks, we will highlight the advances in disk, flash, tape and storage management that aim to keep pace with this growing demand and prevent Stor-mageddon.
Photo courtesy of Shutterstock.