Building an Internal Storage Service Utility

A growing trend among large corporations is to set up their IT departments as distinct business units that sell their services to the other business units of the company.

AOL has taken this a step further by setting up a storage service utility. The various departments buy storage capacity by the GB as well as additional storage services via an interesting chargeback system that is used in the provisioning of several terabytes (TB) every week.

The company has organized its storage staff to facilitate this system. Storage systems administrators deal with storage operations such as provisioning. Capacity managers interact directly with customers (the various lines of business), and programmers develop various storage and Web-based tools.

According to AOL senior storage manager Dan Pollack, the company uses an array of storage discovery tools to manage over three and a half petabytes (PB) of data.

“In-house tracking tools are used for asset management tracking to work out costs on a monthly basis,” says Pollack. “Onaro SANScreen and an in-house Fibre Channel fabric performance tool are used to see how the environment is operating right now.”

For example, these tools can pick up events such as a component being plugged in incorrectly. In addition, AOL EMC ECC is used in the discovery of arrays, what they are called and which ones are in use, and Computer Associates SAN Designer is utilized for visual documentation.

The company finds it best to make host bus adapters (HBA) fit with the existing host and operating system rather than selecting an HBA and trying to make it fit in. It also limits the number of supported storage systems and operating systems for the sake of simplicity and economy.

“We evaluated our technology every six months,” says Pollack. “Each new generation has to be cheaper than the last generation as well as better.”

Developing SLAs

Prior to the rollout of its end-to-end storage chargeback system, AOL struggled to show management how much value it gained from various IT expenditures. According to Pollack, it took a year to show real savings on some storage projects. This was not so much due to low ROI as it was to lack of administrative processes to record costs and returns.

Accordingly, the company established service level agreements (SLAs) for its internal customers to facilitate chargeback administration. One important aspect of provision, for example, was to make it easy to understand the expectations for delivery of storage. That way, no questions would linger about how to deliver, measure or charge for different services.

Another SLA is that once connectivity has been established, it can take no more than a week to provision up to 1 TB. Any provisioning greater than 1 TB, however, is granted a period of up to 90 days, depending on the location and storage availability.

When it comes to availability SLAs, Pollack says AOL is still figuring out that aspect. In terms of performance SLAs, AOL has adopted a policy of having performance be application-driven rather than user-driven. Otherwise, everybody demands the highest levels of bandwidth.

“We have to be stringent on performance needs or everybody asks for the maximum,” says Pollack.

The organization has developed a detailed provision process that is continually evolving. In the three and a half years since the chargeback system began, provisioning has become far more sophisticated.

System Reservations

As the system rolled out, AOL saw the need to develop yet more tools to aid in managing the immense scope of its storage environment. A reservation tool is used to request specific amounts of storage capacity for the various lines of business. A forecasting tool lets storage administrators see the demand for capacity, how it is growing, how much storage resources are unutilized and what storage will need to be added to keep up with expansion.

As a further refinement in chargeback administration, AOL can view costs and storage usage for the lines of business as a whole. Additionally, administrators can drill down into each line of business to view the various functional areas that comprise it. This makes it easy to see who is using their storage assets wisely, and most importantly, who should be billed what.

“Before, people would demand storage and get it with no accountability,” says Pollack. “Now we can bill each area for exactly what they use.”

Article courtesy of Enterprise IT Planet