Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
With two layers, the parallel file system was the primary storage for the system. Any data that needed to be kept for a longer time but wasn't accessed very often was sent to tape (archive). Tools that automatically performed the migration between the two tiers of storage were, and are, fairly common. There are tools and techniques for making all data, regardless of whether it's on tape or disk, appear to be on the same file system (e.g. HSM - Hierarchical Storage Management). The has tremendous advantages for the user because the same set of commands can be used on a file regardless of whether it's in the parallel file system or in the archive.
John Bent, who has worked at Los Alamos for many years on innovative storage systems, predicts that several of the storage layers will ultimately collapse into a single layer. He illustrates this in the diagram below.
Specifically he sees parallel file systems, object storage ("campaign storage") and archive storage collapsing down into 1-2 layers. This brings the number of storage layers down to 2-3, which is better than four.
The top storage layer is NVRAM (burst buffers) that are inside the nodes. The next layer down can either be parallel file systems or a combination of parallel file systems and object storage. The final and third storage layer is either an archive or a combination of an archive and object storage (recall that the Bent says that the parallel file system, object store and archive are to be split into two tiers).
New Archive Media
Traditionally, archive meant a storage layer where you place data that is infrequently accessed but still has to be available to be read. The data is written to the archive layer in a sequential fashion, and there is really no such thing as random access because the data is to be accessed very infrequently. The classic solution for this has been tape.
Today tape is commonplace. It has high density, several tape solutions have very large capacities, and the media is stable and reliable. However, the needed tape robots are expensive and generally have high maintenance costs. For archive data they are an obvious choice versus storing everything on spinning media (hard drives). But there is some new technology that might change things.
Recently, there was an article about storing data in five dimensions on nanostructure glass that can survive for billions of years. This comes from the University of Southampton, where researchers have developed a method of using lasers to read and write to a fused quartz substrate (glass). Currently they are capable of writing 360TB to a 1-inch glass wafer. These wafers can withstand temperatures of up to 1,000 deg. Celsius and are stable at room temperature for billions of years (13.8 Billion years at 190 deg Celsius).
The technology is still being developed and commercialized, so many aspects of it are unknown. The read and write speeds are unknown, but it is a fair assumption that the data is written to the glass wafers in a sequential manner, and random IO is not allowed (sounds a great deal like tape). But the promise of the technology is massive. The researchers have already written several historical documents to a wafer as a demonstration. Such a dense and stable media is an obvious solution for archiving data.
Gunfight at the Storage Coral
The burst buffer storage layer uses NVRAM for storage, and the archive layer either uses tape or most likely, a new media such as the glass wafers previously mentioned. The two middle layers of parallel file system such as Lustre, and the object storage layer, are where data needs to be accessed in a random manner including random write access and re-writing data files. These two layers are the only places where classic storage media such as hard drives or SSDs could reside.
The capacity of hard drives is continually increasing with manufacturers releasing 8TB and 10TB 3.5" drives. To create these increased capacities, manufacturers have started to use shingled magnetic recording drives (SMR). SMR drives allow the density of the individual platters to be increased at the cost of greatly reduced random access write performance. To write some changed data involves first reading the data from surrounding tracks, writing it to available tracks, and then writing the changed data to the drive. Consequently, re-writing data is a very time-consuming process. This has led people to refer to SMR drives as "sequential" drives. This also sounds a great deal like archive storage.