Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Configuring storage area networks (SANs), even at the enterprise scale, is traditionally undertaken by human experts using a time-consuming process of trial and error, guided by simple rules of thumb. Due to the complexity of the design process and lack of workload information, the resulting systems often cost significantly more than necessary or fail to perform adequately.
This article presents a solution to this problem. It shows you how you can automate the design and configuration process through the use of various tools to explore the design space more thoroughly and automatically implement the design, thereby eliminating many tedious, error-prone operations.
For example, one of these tools, from Hewlett-Packard, is structured as an iterative loop design tool (ILDT) for storage area networks (SANs): it analyzes a workload to determine its requirements, creates a new SAN design to better meet these requirements, and migrates the existing system to the new design. The tool also repeats the loop until it finds a SAN design that satisfies the workload's I/O requirements. This article describes the iterative loop and demonstrates (via a discussion) a prototype implementation that converges rapidly to appropriate SAN designs.
Running Circles Around SAN Administration
Enterprise-scale SANs are extremely difficult to manage. The size of these systems, the thousands of configuration choices, and the lack of information about workload behaviors raise numerous management challenges. Users' demand for larger data capacities, more predictable performance, and faster deployment of new applications and services exacerbate the management problems. Worse, administrators skilled in designing, implementing and managing SANs are expensive and in short supply. It is estimated that the cost of managing a SAN is several times the purchase price of the storage hardware. These difficulties are beginning to cause enterprise customers to out-source their storage needs to Internet data centers and storage service providers (such as Exodus), who will lease a SAN. The growing importance of this storage model implies that the ability to accurately provision SANs to meet workload needs, will become even more critical in the future.
Storage management challenges include designing and implementing the storage system, adapting to changes in workloads and device status, designing the SAN, and backing up the data. Thus, with the preceding in mind, you need to concentrate on the important problem of SAN configuration: designing and implementing the SAN needed to support a particular workload, before the SAN is put into production use.
Given a pool of storage resources and a workload, you will want to determine how to automatically choose storage devices, determine the appropriate device configurations, and assign the workload to the configured storage. These tasks are challenging, because the large number of design choices may interact with each other in poorly understood ways. To make reasonable design choices, administrators need detailed knowledge of applications' storage behavior, which is difficult to obtain. Once a design has been determined, implementing the chosen design is time-consuming, tedious and error-prone. A mistake in any of the implementation operations is difficult to identify, and can result in a failure to meet the performance requirements of the workload. SAN configuration is naturally an iterative process, traditionally undertaken by human experts using rules of thumb gained through years of experience. They start with a first design based on an initial understanding of the workload, and then successively refine the design based on the observed behavior of the system.
Unfortunately, the complexities of the systems being designed, coupled with inadequate information about the true workload requirements, mean that the resulting systems are often over-provisioned, so that they are too expensive, or under-provisioned, so that they perform poorly.
For example, the iterative loop design tool (ILDT) automates the iterative approach to SAN configuration. ILDT analyzes a running workload to determine its requirements, calculates a new SAN design, and migrates the existing system to the new design. The ILDT makes better design decisions by systematically exploring the large space of possible designs. ILDT also decreases the chance of human error by automating the configuration tasks. As a result, ILDT frees administrators to focus on the applications that use the SAN. In other words, ILDT generates SAN configurations that employ near minimal resources to satisfy workload requirements, and that it converges to the final system design in a small number of iterations.
The iterative approach to system management is applicable to many levels of the system, including the block-level array subsystem, the file system and the application itself. Therefore, you need to focus on the block-level storage, as it provides a potential benefit to all applications that store data, including those that use the file system and those that use the raw block interface directly. The three stages of the iterative storage management loop as follows:
Design New System
You need to design a system to match the current workload requirements. This stage includes choosing which storage devices to use, selecting their configurations, and determining how to map the workload's data onto the configured devices. The requirements may come from observations of the workload behavior in previous iterations. In other words, a workload is the set of requests observed by the SAN. A particular workload may be generated by one or more applications using the SAN. A workload can be described in terms of stores and streams. A store is a logically contiguous chunk of storage. A stream captures information about the I/O accesses to a single associated store, such as average request rate and average request size. Expressing a workload in terms of stores and streams, decouples the specification of the workload from the application(s) that generate that workload. As a result, the workload specification and assignment techniques are applicable to a broad range of applications.
You also need to configure the disk arrays and other SAN components. In addition, you need to enable access to the storage resources from the hosts, and migrate the existing application data (if any) to the new design.
Finally, you need to analyze the running system to learn the workload's behavior. This information can then be used as input to the design stage in the next iteration.
Removing the Human Administrator Factor
You will want to remove the human administrators from the loop as much as possible, by automating the iterative loop, to the point where all that is required at the beginning is workload capacity information. The loop will then learn the performance requirements across multiple iterations of the loop.
In order to be considered successful, the automated loop must meet two goals. First, it must converge on a viable design that meets the workload's requirements without over- or under-provisioning. Second, it must converge to a stable final system as quickly as possible, with as little input as possible required from its users.Now, let's look at the components of the ILDT loop in more detail. Also, an explanation will be given on how the components interact to design a SAN iteratively.