Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
NAS solutions can provide reasonable performance for a wide range of applications in the cloud and allow you to run many client instances sharing the same storage (shared storage). But what happens if you need more performance than what a single NAS gateway can provide? Or what do you do when you have applications that can benefit from parallel IO? The good news is that there are some solutions available in the Amazon Marketplace. These AMIs provide shared parallel storage for applications.
Parallel storage allows you to combine instances (i.e. more than one) and storage to a single file system. This allows very large capacities and faster throughput. The exact details of how the storage and servers are combined into a single file system depend upon the specific file system, but conceptually the idea is to spread portions of the file across multiple servers with their own storage so that the data access for a file can be done in parallel and so that you can lose a server and storage without loss of data (or access to the data).
Lustre: Lustre is the most common parallel file system for HPC according to some studies (e.g. 2013 IDC file system study). It's open-source with a development community and two sponsoring groups (Open Scalable File System - OpenSFS and European Open File System - EOFS). There are several companies that will sell either support contracts for Lustre (e.g. Intel) or a combined hardware and software solution (Xyratex, DDN, Dell, Terascala, Bull, Inspur) that includes support for Lustre in addition to the hardware.
Recently, Intel has created a version of Lustre that runs in AWS using EC2 instances and EBS volumes. This version, called ICEL(Intel Cloud Edition of Lustre) has three different instance options at this time, one of which is free to use but doesn't come with any support. The instances are:
The two versions with Global Support provide you with support, one of which uses HVM instances. The "Community Version" is free to use, but if you run into problems, you have to turn to the community Lustre mailing lists for help.
Just like other storage solutions in the cloud, ICEL is constructed from AWS compute and storage instances. You use an instance as an OSS (Object Storage Server) and attach EBS volumes to it for storage. You do the same for the MDS (Metdata Server) and MGS (Management Server) servers: select a compute instance and then attach EBS volumes to it. To get the performance desired, you select the number of OSS instances you need and to get the capacity you want, you attach a number of EBS volumes to each OSS. (To make things symmetrical, it is recommended that you use the same number of EBS volumes for each OSS instance.)
Assembling the OSS and MDS instances with attached and formatted EBS volumes including installing and configuring Lustre can be time consuming. ICEL has created a CloudFormation template that does all of this for you. You only need to set the KeyName value to your ssh-key, and set the LustreZA value to the specific Availability Zone in the region you are using, and the template automates the deployment of ICEL for you. Once ICEL is up and running, you then have to install the Lustre client on the instances you plan to use with ICEL.
There are some materials around the web that discuss ICEL and the performance. The first presentation was in April 2013 at the 2013 Lustre User Group (LUG). There is also a video of the paper, where Robert Read discusses Lustre in AWS. Notice that Robert did some testing using DNE (Distributed NameSpace) for Lustre, which is the first release of distributed metadata for Lustre. The really amazing thing about building parallel storage solutions in AWS is that if you need more aggregate IO throughput, you can just add OSS instances. If you need more capacity, you just spin up additional OSS nodes. If you need more aggregate metadata performance, you just spin up MDS instances with EBS volumes. You don't have to wait for weeks to get new hardware—you just spin up a new instance, and you are off to the races.
The most recent presentation is from the Fall of 2013 at the LAD13 conference (European Lustre User Group Conference). The ICEL presentation is available online. There is an accompanying video of the presentation as well. If you notice in the presentation, the authors mention that the current Lustre AMIs in Marketplace come with Ganglia, LMT (Lustre Monitoring Tool) and ltop (part of LMT), which can be used for monitoring the Lustre file system. They even have a screenshot of Ganglia illustrating monitoring tools for ICEL.
As a matter of full disclosure, I work at Intel as the senior product manager for Intel Lustre in the High Performance Data Division (HPDD), but the ICEL product is predominantly handled by another team with the division.
OrangeFS: One of the very first parallel file systems is PVFS (Parallel Virtual File System). It started as a research project at Clemson University with the first paper being published in 1996. PVFS has been steadily refined and developed over time, even spawning a rewrite, PVFS2. A software company named Omnibond has recently created a commercial version of PVFS2, called OrangeFS that comes with full support.
There is a version of OrangeFS in AWS that uses the compute instances as well as the EBS volumes. "OrangeFS in the Cloud" was announced in May of 2013. There are two AMIs:
The community version does not use provisioned IOPS with the EBS volumes, resulting in lower performance. There are a total of seven possible EC2 compute instances you can use with this AMI. It also has a fixed price per instance of $0.45/hr but no support.
The Advanced Version comes with support, costs $0.85/hour, and uses the same instances as the Community Version except that it uses Provisioned IOPS to achieve better IOPS performance.
Both the community and advanced editions have four creation options:
- "1-click" single instance with 1.28TB of capacity
- 4 instances with 5 TB of capacity
- 8 instances with 10TB of capacity
- 16 instances with 20 TB of capacity
The number of instances refers to the number of EC2 compute instances used.
There are a couple of references on the web that talk about OrangeFS itself and about using OrangeFS in the cloud. You can read about PVFS or OrangeFS at their respective websites but there is a fairly extensive paperfrom the 2012 IEEE International Conference on Massive Storage Systems and Technology that is worth reading. It is authored by by Walt Ligon, who was one of the originators of the PVFS project.
A second item of note is a presentation entitled, An MPI-IO Cloud Cluster Bioinformatices Summer Project. The presentation is quite interesting because it has a couple of parts. The first part is a presentation on the Amazon storage components. The second part is about using bioinformatics applications that use MPI-IO and OrangeFS in the cloud.
More and more technical computing or HPC workloads are moving into the cloud. Many of these workloads need or work better with shared storage. Creating shared storage in the cloud is only beginning to be addressed. These solutions are built from the same components as NAS solutions in your data center: compute, network, storage and software. The big difference is that you can spin up a compute instance or add storage in the matter of a few minutes rather than weeks or perhaps months waiting for the hardware to arrive, be installed, tested and put into production.
Using Amazon as an example, there are some initial designs and experiments with NAS in the cloud including some commercial solutions such as SoftNAS Cloud. It's fairly easy to configure your own NAS solution if you desire using the exact same commands you would use if you were building your own solution. Commercial solutions such as SoftNAS Cloud offer the option of having all of the heavy lifting done for you in exchange for paying for the software (and you get support).
In addition to NAS, you can also create parallel storage solutions. For example, in Amazon AWS, there are two options, one for Lustre, and one for OrangeFS (PVFS). Both use the same compute and storage instances that you use for NAS, but you create several instances that are combined to create a single file system. If you need more performance, just add more instances. If you need more capacity, just add more instances. Since this is the cloud, it's very easy to spin up a new instance and add it to the existing storage.
Photo courtesy of Shutterstock.