Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
What Will Our Future Look Like?
We have gotten to a point where I think the amount of storage complexity exceeds the volume of storage talent. Combine this with the facts that we still do not have a common management framework, and that we have new applications, appliances and methods. I think we are seeing a rise of appliances which do not require high-end storage administrators, except at extreme scale.
For example, most of the parallel file system community has moved to storage appliances which have few knobs and switches. Most of the purchasing community for HPC environments has quickly embraced this technology, given the high cost and long training time needed to administer these file systems.
So if you are a skilled, highly talented administrator, what should be your plan to ensure that your salary does not take a nose dive?
I think the answer is appliances for data analysis. (I am likely not talking about Hadoop, as many of the architectural designs for products in this area are completed.) Data analysis appliances are in their infancy today and will require significant care and feeding. The types of data analysis are going to be very complex. For example, you might de-pixelize an image and create a database of geolocations, normalizing for the resolution of the image, which might change over time based on improvements in technology. Then you might correlate the pixels to look for weather, climate or some other change like deforestation. This will be far different than taking business data and trying to correlate prices to sales to maximize profits.
Things that used to be difficult are going to be easier. But I suspect there will be a new class of more complex appliances that will try to address a wide variety of problems, and they are going to require significant tuning and configuration. The information collected and processed will have to be architected so that access is efficient when trying to correlate the information, process it and provide results to decisions makers.
The storage complexity problem for file systems has been mostly solved. There are still a few hard problems out there, but not as many as there used to be.
However, there is a new and even more complex set of problems right in front of us. These will require a deep understanding of what the users need to do with the storage and how they plan to access the data to create information on which actionable decisions can be made. These jobs are going to be high paying and require a broad set of skills. But the skills will be different than the current skills required for SAN and NAS and even the other types of appliances that are out there. Those involved are going to have to work directly with the application developers and users.
Come to think of it, this sounds a great deal like 1996 and 1997 when SAN file systems started to come out. Those of us involved then had to talk with everyone up and down the stack to get things going quickly and efficiently. I believe the same approach is needed today.