Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Broader adoption of machine learning and artificial intelligence (AI) has some system and storage managers really excited. Machine learning algorithms, for example, can be incorporated into the control layer to enable administrators to diagnose the various causes of traffic congestions far more easily. This allows them to predict potentially vulnerable network sectors.
“User requests and data traffic can be channeled to and from alternative storage locations based on network usage patterns,” said Shiladitya Chaterji, an AI analyst at MarketsAndMarkets.
But it goes far beyond being a mere traffic cop. AI and machine learning are influencing data storage in many different ways. Here are some of the top trends and predictions:
1. Lower Costs
With enterprises moving towards cloud storage and fewer dedicated storage arrays, dynamic storage software with integrated deep learning algorithms can help organization to gain more storage capacity, at a 60 percent to 70 percent reduction in cost,” said Chaterji.
2. More Software-Defined Storage
Software-defined storage has been touted as a trend for a couple of years now. AI and machine learning are acting as accelerants. The many potential benefits are helping enterprises overcome their reticence to being told there is yet another new technology to adopt.
“AI and machine learning will bring about faster adoption of software-define storage,” said Kevin Liebl, vice president of marketing at Zadara Storage.
3. More Instrumentation
The early days of computing saw plenty of instrumentation being added to systems. In fact, there are entire conferences and associations devoted to the measurement and instrumentation of computers.
With Windows servers proliferating from the mid-1990s onwards, this side of the business has gradually waned. But that appears to be changing as AI and machine learning open up new horizons. Consequently, Liebl predicts another trend of much greater instrumentation in the years ahead.
The advent of software-defined storage is a key influence in the rise of machine learning and AI in storage environments. Adding a heterogeneous software control layer above the hardware allows the software to monitor far more tasks. This frees up the storage manager for more strategic duties.
“AI can enable automation of storage facilities that adopt an agile and flexible architecture,” said Chaterji. “It can intelligently control access rights, dynamically re-route data center data and automatically regulate data center cooling (thereby reducing energy consumption).”
5. Enhanced Security and Reliability
Security and loss of are major concerns for the modern enterprise. Some storage vendors are beginning to harness AI and machine learning to prevent data loss, increase availability and speed turnaround during downtime via smart data recovery and systematic backup strategies, said Chaterji. He added that this also promises better security.
“AI opens the door to smart security features to detect data/packet loss during transit or within data centers,” said Chaterji.
6. Hybrid Storage Clouds
The argument about public versus private clouds appears to be moot in the face of AI, machine learning and software-defined storage. That's because functional software-defined architectures should be able to transition data seamlessly from one type of cloud to another. At the same time, organizations can manage all their data as one pool, regardless of where it physically resides. As a result, the purists who seek all public or all private clouds are not likely to prevail. It is the hybrid cloud that is most likely to flourish.
“The use of artificial intelligence and machine learning will accelerate the deployment of fluid hybrid cloud solutions as a repository because after data is analyzed and logic maps are developed, they must flow transparently to local analytics engines at the edge in a cycle of continuous improvement,” said Rich Rogers, senior vice president of Internet of Things (IoT) products and technologies, HDS.
7. More Flash
Everyone predicts more flash, so what’s new? AI and machine learning will just add yet more impetus to this almost unstoppable wave that is sweeping across all forms of storage.
“They will drive the use of memory and flash as a medium as the primary storage medium, as you cannot process edge decisions fast enough otherwise,” said Rogers.
8. Automated Cars
Perhaps the biggest driver that will ultimately provide the use case for integrating AI and machine learning into storage will be drivers — car drivers. Today’s high-end cars (without autonomous features) have anywhere between 64 and 200 GB of storage — mostly for maps and infotainment functions. In tomorrow’s autonomous vehicles, we might see more than 1 TB of storage, and this will not just be for the drive function.
“Intelligent assistants in the car, advanced voice and gesture recognition, caching software updates and buffering infotainment to reduce peak network bandwidth utilization will all be driving factors of why more storage will be required locally,” said Martin Booth, director of marketing, automotive solutions, Western Digital.
9. Parallel File Systems
In order to support AI and machine learning capabilities, storage systems will have to deliver performance at scale. This means they must be able to work well at projected scale with technologies like parallel file systems and flash. So said Laura Shepard, senior director, product marketing, DataDirect Networks.
“For future-proofed infrastructure, the system also needs to be able to simply and cost-effectively handle data that is older or colder and support a clear path to future technologies like new flash formats and flash-native tools that maximize flash performance while avoiding flash-specific performance and longevity hurdles,” she said.
10. Neural Storage
Liebl also forecasts the rise of “neural-class” storage. This is where the storage can recognize and respond to problems and opportunities without human intervention. When that technology takes hold, expect a step change in productivity.
Frank Berry, an analyst at IT Brand Pulse, said arriving at neural storage won’t happen overnight. He laid out three phase culminating in the realization of neural storage networks. They will manifest gradually and one will lead to the other. Phase 1 is, as mentioned by Liebl, where storage is instrumented with telemetry to collect data from non-traditional sources. For example, user-level access patterns, data flows, networking flows, and data about hardware and software failures. This phase manifests in the relatively early stages of software defined storage.
Phase 2 is what Berry refers to as self-driving. Once storage is all software-defined, which algorithms can become integrated and far-reaching enough to solve complex storage management problems courtesy of the wealth of new data they can access. This is a necessary step on the road to building the monitoring, tuning, healing service chains needed for self-driving.
Only when those two phases have been attained, can neural storage networks take root.
“True neural networking (layers of processing with mountains of data) is integrated into storage infrastructure which allows it to learn and develop new capabilities on its own,” said Berry.
In some ways, this may be the stuff of science fiction. HAL (from the movie "2001 Space Odyssey") came to the logical conclusion that his crew had to be eliminated. Perhaps neural storage will come to the conclusion that 99.99999 percent of stored data has no value and so should be deleted. But no doubt, some good will come of this neural storage concept.
Photo courtesy of Shutterstock.