Implementing RAID in Enterprise Environments

A great deal has been written about RAID hardware, software, and a myriad of related topics since the early 1990s when hardware RAID first appeared on the scene. This month I’ll present the topic of RAID from a slightly different perspective than what you’ve likely previously seen written.

Types of RAID

I separate RAID devices into two types. They are:

1. Cache-centric RAIDs
2. Storage-centric RAIDs

Often these two types of technology are called “enterprise RAID” (cache-centric) and “mid-range RAID” (storage-centric). Whatever you choose to call these devices, there are major structural differences between the two types, even though both are called RAID and perform many of the same functions.

Cache-centric RAIDs

I use the term cache-centric because RAID devices in this category depend significantly on data residing in cache to achieve good performance. Most cache-centric devices generally have feature sets such as:

• Very high reliability (dual and hot swap everything, with virtually no downtime)
• Large caches (64 GB or greater)
• Designed emphasis on using RAID-1
• Software that allows for snapshots, hot upgrades, and a plethora of other features
• If RAID-5 is supported, generally a smaller number of devices-supported stripe sizes (e.g. 3+1 — 3 data disks plus 1 parity drive — or 7+1)
• Cache is always synchronously mirrored
• Large number of front-end connections
• Support for many types of remote mirroring (dark fibre, IP)
• Smaller block sizes are more typical (4 KB, 8 KB)
• Huge amounts of storage managed in a single box (100 TB)
• Great deal of per component reliability testing
• Error monitoring for hardware
• Error monitoring for each disk for soft errors such as write and read retry and remapping of sectors
• Designed for IOPs (Input/Output Processors), not streaming I/O
• Far more bandwidth from cache to the servers than from cache to disk (sometimes up to 4 times greater). This is often called front-end bandwidth (cache to servers) and back-end bandwidth (cache to disk)
• Much higher cost per MB of storage compared with storage-centric RAID (not surprising given the reliability and size of the cache)

Some of the cache-centric RAIDs vendors and devices include:

• IBM Shark
• EMC Symmetrix and new DMX2000
• Hitachi Data Systems 99xx series

All of these products can run both on UNIX servers and on IBM mainframes. They are designed to support environments where reliability is an issue and, in some cases, where customers are willing to trade off some performance for reliability. For these RAID products to have great performance they need to have a high number of cache hits. These RAID systems allow huge amounts of storage behind a single controller for simplification of management of the storage.

Page 2: Storage-centric RAIDs

Storage-centric RAIDs

I use the term storage-centric because these devices primarily depend not on the cache but rather on the underlying storage and architecture. These devices have the following features:

• Good reliability, but nowhere near as reliable as the cache-centric devices
• Smaller caches (usually 2GB to 8GB)
• RAID-5 support with up to 20 disks in a RAID-5 LUN (Logical Unit Number)
• RAIDs allowing cache mirroring can often be turned off, wherein significantly improved write performance is realized
• Far less software for management and maintenance
• Far less storage in a single box than enterprise boxes support
• Support for streaming I/O with large blocks (which is not supported well by Enterprise RAIDs)
• Support for high IOPs, but with the smaller caches, most I/O has to be to disk
• More back-end bandwidth than front-end bandwidth (more channel bandwidth from cache to disk than from cache to the servers)
• Much lower cost per MB for storage than cache-centric RAIDs

Examples of storage-centric RAID vendors and products include:

• EMC CLARiiON CX line (Dell resells EMC)
• LSI 5600 (OEM’ed by many other vendors)
• Sun T3 and S1
• Hitachi Data Systems 9500
• DotHill
• DataDirect Networks
• Ciprico
• Many others

There are many vendors in this market area as it is much easier to develop products than for the cache-centric devices given the lower reliability requirement. Additionally, vendors do not have to create mainframe interfaces and the market has a lower expectation for software. Most vendors in this market space are now looking at options that use ATA drives instead of SCSI drives. They are doing this to provide greater data density and lower price points than are available from SCSI drives. Of course, you are trading reliability as bit error rates for Fibre Channel or SCSI drives are an order of magnitude greater than those for ATA drives (according to the Seagate website). RAID devices of this type are far more prevalent and usually have two or more times more bandwidth to disk than to the front-end servers.

Page 3: So Which Is Better?

So Which Is Better?

As with many things in computing, “It depends on many factors, issues, and requirements.” About a year and a half ago, my company was asked to work on a project where the customer wanted to use enterprise storage for capturing high-speed data streams. Internally, we called this project BFASP— “blood from a stone project.” We knew, as did many of the other technical people involved, that the customer could not use an enterprise storage box for high-speed, full duplex I/O (writing files and reading other files at the same time).

The enterprise storage vendors benchmark team said they could do this type of I/O, but our technical team thought this was absurd. After spending 36 hours with them, the benchmark team finally came to this conclusion: “Oh, we did not understand you needed full duplex I/O at these speeds; we cannot do this.” The customer ended up purchasing a storage-centric RAID, and the enterprise storage vendor left with their tails between their legs, as they should have, given the time and money that was wasted.

On the other hand, the same enterprise storage box will far exceed the performance of the storage-centric device for a database where the index files are used often and fit into the cache. A few new vendors are entering the market and claiming to combine the best of both products. Vendors such as EMC with its new DMX2000, 3ParData, and others will be seeking to prove their products can ably perform both functions.

The decision on what to buy is quite often far above your pay grade, but assuming that you do have some say, or at least have some input, you need to review your requirements and your operational usage. Here are just a few of the major issues that you should consider:

1. What are the uptime and reliability requirements?
2. What are your backup requirements?
3. What RAID level will you be using?
4. How large are your files typically, and how are they accessed?

Reliability

It comes down to how much downtime you can afford for the box. If you have many attached hosts, you cannot afford any downtime. This is often called the 9 count, or how many “9s” of reliability are supported. Here’s a chart showing several different combinations of “9s” and their corresponding downtime:

Knowing the reliability requirements and the number of attached hosts will help determine the type of equipment needed.

Page 4: Backup

Backup

Enterprise boxes have the ability to create a mirror, break the mirror for a backup, and then reattach the mirror and update the box. Most mid-range boxes do not have this feature, although it is starting to become more common. Backup is becoming more difficult given the total amount to storage that needs to be kept. More and more often, I am seeing sites move to hierarchical storage management (HSM) techniques rather than backup, given the immense data volumes. See my article on tapesfor additional information on this topic. The issues for backup will be addressed in a few months.

RAID Level

You need to determine what RAID level you are going to need to use. There are many different RAID levels, but the two used most often are RAID-1 and RAID-5. The RAID level depends on:

1. Cost — For the same amount of data storage, RAID-1 requires far more disks than RAID-5.

1. How the data is used — If you are making small random requests, RAID-1 is faster than RAID-5. In general, if you are making large sequential block requests, RAID-5 is faster, especially when you have a large number of sequential writes.

So, if you are making small requests (especially if they are random), RAID-1 will be much faster than RAID-5. With RAID-1 each device is mirrored, but with RAID-5 you create a LUN (Logical Unit Number) with a parity drive so that the LUN can be rebuilt if a device fails. With RAID-1 you write far more data (as each disk is mirrored) than you write with RAID-5, resulting in far more backend bandwidth for sequential I/O.

File Sizes and Accesses

In today’s world, the likelihood of everything fitting into the RAID cache is very low; in fact, why would you even buy a RAID if that was the case, as you could simply purchase an SSD (solid state storage). The real questions you need to ask yourself are how are these files accessed and can the data be reused. This leads to the next important question — would a large cache help or not make any real difference?

Conclusions

The choice between cache-centric RAID devices and storage-centric RAID devices likely will be made by budgetary constraints and other issues rather than the performance of the devices. The operational environments of the two types are often vastly different, with issues to consider including:

• How many LUNs and how much storage do you want under the controller of one device
• What RAID levels do you want to run based on the cost per MB
• What type of disk devices do you want
• What RAID levels are going to provide the best performance given the applications
• The application types and request sizes

What it really comes down to is something I personally always say it comes back to: “The requirements.” Purchases should be made in terms of the requirements and budget so you get the best value for the available funds.

In the next article, we will dig deeper into RAID and discuss how the layout of RAID and file systems should be architected.

» See All Articles by Columnist Henry Newman