Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
A great deal has been written about RAID hardware, software, and a myriad of related topics since the early 1990s when hardware RAID first appeared on the scene. This month I'll present the topic of RAID from a slightly different perspective than what you've likely previously seen written.
Types of RAID
I separate RAID devices into two types. They are:
- Cache-centric RAIDs
- Storage-centric RAIDs
Often these two types of technology are called "enterprise RAID" (cache-centric) and "mid-range RAID" (storage-centric). Whatever you choose to call these devices, there are major structural differences between the two types, even though both are called RAID and perform many of the same functions.
I use the term cache-centric because RAID devices in this category depend significantly on data residing in cache to achieve good performance. Most cache-centric devices generally have feature sets such as:
- Very high reliability (dual and hot swap everything, with virtually no downtime)
- Large caches (64 GB or greater)
- Designed emphasis on using RAID-1
- Software that allows for snapshots, hot upgrades, and a plethora of other features
- If RAID-5 is supported, generally a smaller number of devices-supported stripe sizes (e.g. 3+1 -- 3 data disks plus 1 parity drive -- or 7+1)
- Cache is always synchronously mirrored
- Large number of front-end connections
- Support for many types of remote mirroring (dark fibre, IP)
- Smaller block sizes are more typical (4 KB, 8 KB)
- Huge amounts of storage managed in a single box (100 TB)
- Great deal of per component reliability testing
- Error monitoring for hardware
- Error monitoring for each disk for soft errors such as write and read retry and remapping of sectors
- Designed for IOPs (Input/Output Processors), not streaming I/O
- Far more bandwidth from cache to the servers than from cache to disk (sometimes up to 4 times greater). This is often called front-end bandwidth (cache to servers) and back-end bandwidth (cache to disk)
- Much higher cost per MB of storage compared with storage-centric RAID (not surprising given the reliability and size of the cache)
Some of the cache-centric RAIDs vendors and devices include:
- IBM Shark
- EMC Symmetrix and new DMX2000
- Hitachi Data Systems 99xx series
I use the term storage-centric because these devices primarily depend not on the cache but rather on the underlying storage and architecture. These devices have the following features:
- Good reliability, but nowhere near as reliable as the cache-centric devices
- Smaller caches (usually 2GB to 8GB)
- RAID-5 support with up to 20 disks in a RAID-5 LUN (Logical Unit Number)
- RAIDs allowing cache mirroring can often be turned off, wherein significantly improved write performance is realized
- Far less software for management and maintenance
- Far less storage in a single box than enterprise boxes support
- Support for streaming I/O with large blocks (which is not supported well by Enterprise RAIDs)
- Support for high IOPs, but with the smaller caches, most I/O has to be to disk
- More back-end bandwidth than front-end bandwidth (more channel bandwidth from cache to disk than from cache to the servers)
- Much lower cost per MB for storage than cache-centric RAIDs
Examples of storage-centric RAID vendors and products include:
- EMC CLARiiON CX line (Dell resells EMC)
- LSI 5600 (OEM'ed by many other vendors)
- Sun T3 and S1
- Hitachi Data Systems 9500
- DataDirect Networks
- Many others
So Which Is Better?
As with many things in computing, "It depends on many factors, issues, and requirements." About a year and a half ago, my company was asked to work on a project where the customer wanted to use enterprise storage for capturing high-speed data streams. Internally, we called this project BFASP -- "blood from a stone project." We knew, as did many of the other technical people involved, that the customer could not use an enterprise storage box for high-speed, full duplex I/O (writing files and reading other files at the same time).
The enterprise storage vendors benchmark team said they could do this type of I/O, but our technical team thought this was absurd. After spending 36 hours with them, the benchmark team finally came to this conclusion: "Oh, we did not understand you needed full duplex I/O at these speeds; we cannot do this." The customer ended up purchasing a storage-centric RAID, and the enterprise storage vendor left with their tails between their legs, as they should have, given the time and money that was wasted.
On the other hand, the same enterprise storage box will far exceed the performance of the storage-centric device for a database where the index files are used often and fit into the cache. A few new vendors are entering the market and claiming to combine the best of both products. Vendors such as EMC with its new DMX2000, 3ParData, and others will be seeking to prove their products can ably perform both functions.
The decision on what to buy is quite often far above your pay grade, but assuming that you do have some say, or at least have some input, you need to review your requirements and your operational usage. Here are just a few of the major issues that you should consider:
- What are the uptime and reliability requirements?
- What are your backup requirements?
- What RAID level will you be using?
- How large are your files typically, and how are they accessed?
It comes down to how much downtime you can afford for the box. If you have many attached hosts, you cannot afford any downtime. This is often called the 9 count, or how many "9s" of reliability are supported. Here's a chart showing several different combinations of "9s" and their corresponding downtime:
Downtime Per Year in Seconds
Downtime in Minutes Per Year
Downtime in Hours Per Year
Downtime in Days Per Year
Knowing the reliability requirements and the number of attached hosts will help determine the type of equipment needed.
Enterprise boxes have the ability to create a mirror, break the mirror for a backup, and then reattach the mirror and update the box. Most mid-range boxes do not have this feature, although it is starting to become more common. Backup is becoming more difficult given the total amount to storage that needs to be kept. More and more often, I am seeing sites move to hierarchical storage management (HSM) techniques rather than backup, given the immense data volumes. See my article on tapes for additional information on this topic. The issues for backup will be addressed in a few months.
You need to determine what RAID level you are going to need to use. There are many different RAID levels, but the two used most often are RAID-1 and RAID-5. The RAID level depends on:
- Cost -- For the same amount of data storage, RAID-1 requires far more disks than RAID-5.
- How the data is used -- If you are making small random requests, RAID-1 is faster than RAID-5. In general, if you are making large sequential block requests, RAID-5 is faster, especially when you have a large number of sequential writes.
So, if you are making small requests (especially if they are random), RAID-1 will be much faster than RAID-5. With RAID-1 each device is mirrored, but with RAID-5 you create a LUN (Logical Unit Number) with a parity drive so that the LUN can be rebuilt if a device fails. With RAID-1 you write far more data (as each disk is mirrored) than you write with RAID-5, resulting in far more backend bandwidth for sequential I/O.
File Sizes and Accesses
In today's world, the likelihood of everything fitting into the RAID cache is very low; in fact, why would you even buy a RAID if that was the case, as you could simply purchase an SSD (solid state storage). The real questions you need to ask yourself are how are these files accessed and can the data be reused. This leads to the next important question -- would a large cache help or not make any real difference?
The choice between cache-centric RAID devices and storage-centric RAID devices likely will be made by budgetary constraints and other issues rather than the performance of the devices. The operational environments of the two types are often vastly different, with issues to consider including:
- How many LUNs and how much storage do you want under the controller of one device
- What RAID levels do you want to run based on the cost per MB
- What type of disk devices do you want
- What RAID levels are going to provide the best performance given the applications
- The application types and request sizes
In the next article, we will dig deeper into RAID and discuss how the layout of RAID and file systems should be architected.