Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Big File System Appliances
For now there are two different large shared file systems used for large storage appliances—GPFS and Lustre. Multiple vendors make these appliances. While Lustre is an open source project, GPFS is a product from IBM.
These file systems scale far beyond what any current NAS vendor offering has. Both of these file systems scale to thousands of clients and offer performance to many hundreds of GiB/sec. What NAS vendor has 30+ PB in a single namespace that has scalable performance?
- Understand your workload in terms of:
- How many I/O requests are being done at the same time?
- What are the read/write ratio and read and write request sizes?
- How many open/creates being are being done at one time?
- How much storage is needed?
- Is ANSI T10 DIF/PI used?
- Is some other method used?
- Does it use checksums or error correction code?
- How does the vendor tell you which disk drive is causing the errors?
- Is the RAID declustered?
- How long does a rebuild take?
- What is the performance hit during rebuild?
These are just some of the important questions that need to be asked for these types of appliances. As you scale up, you need to ensure that nothing else breaks and that you can meet your mission.
There are lots of other data analysis products and database products that are here today and even more coming down the path that might meet your organization's requirements. These new appliances might correlate information in some method or use graph analysis to look for relationships and/or some other method.
The issues are going to be the same: if you can't do it faster and cheaper locally with the technology you have, then your work might be outsourced to the cloud.
Our Jobs Are at Stake!
If CIOs and the staffs at many companies do not get with the plan, someone else will. And then someone else will do your job for you.
My best friend once said to me when we were in Japan testing a new file system we had designed, "We need to eat sushi or we will be sushi." The point was you either go with the flow of what is happening or you become a statistic.
I think the way things are going in the market, we are all going to have to learn some new skills. Management has to be included in this process, as things are going to have to be organized to be efficient.
If not, be prepared to have your work outsourced either to the cloud or to another organization.
There are a bunch of new technologies that are going to solve new problems and old problems. We all need to become familiar with these technologies to survive.
I think that much of what we see today for storage will become specialized appliances. While 90 percent of the data accessed today is likely accessed via a POSIX file system interface and 10 percent from an object interface, that is going to change over the rest of the decade.
Get ready—this will be similar to the micro-processor ride in the 1990s when we changed from proprietary large processors to micro-processors from DEC, MIPS, Intel and others.