The Internet revolution has changed the way business is conducted around the globe. The most significant revenue opportunities available to enterprises today are enabled by e-commerce and e-business. Implementing storage area network (SAN) hosted database applications effectively allows an organization to realize tremendous return on investment by achieving competitive advantage, process efficiency and organizational streamlining.
Because large-scale e-commerce and e-business operations must be online around the clock, the SAN database applications and underlying infrastructures that enable them must deliver consistent high performance and continuous, scalable operation. For the enterprise SAN infrastructures that support these database applications, this means specifying the appropriate class of product to meet the demands of the application.
Implementing e-commerce and e-business SAN applications effectively requires that business information be made more accessible, scalable and manageable. The implementation of an enterprise SAN provides the consolidated infrastructure and resource-sharing capability necessary to achieve this.
With the preceding in mind, Part I of this two-part article first discusses the organizational considerations when implementing SANs. This article very briefly outlines what experience has demonstrated — what should be done, and who should do it. It does not outline how to accomplish the administrative and management functions of central storage or SAN management, but rather how to organize the personnel and teams that will implement and manage the new storage environment. Part IIcontinues the SAN implementation theme by briefly discussing other SAN implementation topics with regards to backups, clusters, appliances and database applications.
Organizational Issues During SAN Implementation
Enterprises are constantly looking to implement new storage architectures that will allow them to exploit today's storage technologies. Many of these technologies, such as SANs, Fibre Channel (FC), Fibre Channel over IP (FCIP), SCSI over IP (iSCSI), and network attached storage (NAS), have common operational issues, even through they are differing storage I/O path technologies. When moving from direct storage devices to pooled resources, a common element of pooled storage architectures is the change necessary within enterprises to adapt and adjust and therefore maximize their investments.
The separation of the direct attached disk and tape from servers has created some confusion regarding ownership, management, and accountability within enterprises. Certainly, a radical change in storage architecture and access cannot be completed without adjustments to enterprises, roles and responsibilities, management tasks, and criteria for proper execution of duties.
One of the last elements enterprises tend to consider as part of SAN implementation, is the impact to the organization. This article briefly discusses technical and operational aspects of pooled storage or SAN technology, and the impact this will have on the enterprise. Most of the risk areas can be mitigated if the organizational impact is considered early on as part of the SAN design and implementation program.
Several variables impact the storage team's organizational structures directly. These include (but are not limited to):
- Acceptance of change within the enterprise.
- Business model type in use for the enterprise and the lines of business (shared services Information Technology (IT), distributed IT services, outsourced IT functions).
- Centralized versus decentralized support model.
- Chargeback methods and other financial aspects.
- Current state of enterprise maturity with regard to general enterprise best practices, skills, and organization.
- Expectations from the new pooled storage or SAN infrastructure.
- SAN topology and implementation decisions.
- Staff skills and geographical locations.
It is important to realize that (as with all technologies) there is a time-dependant variable in the impact to the staff and organization, when describing organizational dynamics relative to storage. Organization and team roles needed during steady state management (where optimization is paramount), are very different from SAN implementation plans during the start-up or installation phase.
SAN Implementation Team Creation
One of the most common questions asked during a SAN implementation is: What kind of organizational structure do you need to support this SAN? This section of the article briefly deals with potential organizational arrangements — structures relative to any SAN plan or pooled storage architecture. There are several dimensions that need to be considered, namely:
- The SAN team organizational reporting structures.
- Alignment of SAN architectural and organizational plans to the appropriate type of operating business model of the enterprise.
- Bi-directional impact of the SAN topology and the resulting management team.
SAN Implementation Team Organizational Reporting Structure
The organizational alignment of a new storage team is one of the early concerns. Most medium-to-large enterprises are organized along platform lines, and thus (up until now) have had responsibility for storage and tape operations that relate to a specific operating system or server platform. The functions of servers and storage/tape can be separated with advanced storage architectures. And, these groups may go through a fundamental change in providing service, accountability, span, and control of enterprise storage resources.
Alignment of SAN Architectural Implementation and Organizational Plans
The basic roles of the storage team are to architect, design, plan, implement, administer, control, and generally provide 'care and feeding' for the elements of the SAN and pooled storage systems. In cooperation with other teams and functions within the IT department, the exact span and control of the environment will have to be sorted by each individual team. A checklist of items within the SAN and pooled storage framework that need to be owned is shown next. Ideally, all elements would come under administrative control of the SAN team:
- Backup, recovery, and disaster recovery (DR) schemes, schedules, and test plans.
- Backup servers, backup managers, APIs resident on slave servers.
- Enterprise storage arrays, Gigabit Interface Controller (GBIC), software.
- Enterprise tape drives, libraries, Active Template Library (ATL), etc.; Vtape, optical libraries.
- External, SAN management software (Logical Unit Number (LUN) mapping, management) for both in-band and out-of-band implementations.
- FC extenders, dense wavelength division multiplexing (DWDM), and other native FC multiplexing or amplification devices used for long-distance extension.
- FC hubs, FC switches, FC directors
- Fibre Channel-asynchronous transfer mode (FC-ATM) or other gateway devices that use the wide area network (WAN) to tunnel FC or otherwise extend FC through WAN access.
- Fibre Channel (FC) cables, plant management, patch panels.
- Fibre Channel/Internet Protocol (FC/IP) or Internet Small Computer Systems Interface (iSCSI) gateway device.
- Fibre Channel-Small Computer Systems Interface (FC-SCSI) bridges.
- HBA utilities and software.
- Host or storage data replication, snapshot, redundant arrays of independent disks (RAID), or data mover software.
- Host volume management software.
- Just A Bunch Of Disks (JBOD) connected to pooled fabric architecture.
- NAS appliance, NAS filer, or other general-purpose server used for Common Internet File System (CIFS) or Network File System (NFS) file serving.
- SAN management software imbedded within switches or directors.
- Server Host Bust Adapters (HBAs).
- Storage virtualization software.
- Switch and director OS, software.
Bi-Directional Impact of the SAN Implementation Topology
A complex undertaking can be designing and implementing some SAN topologies. Before any topology is decided upon, it is important that organizational structure, people skills, and teaming options are well defined. The project will fail if a complex SAN (complex from the point of view that advanced management skills and processes are required) is implemented into an organization that cannot sustain the technology. When deciding on the topology to deploy, a critical analysis of the internal workings of the organization is essential. Also, when defining the new storage management/SAN team, this same analysis is needed.
SAN Implementation Core Team Skills and Team Dynamics
In order to engineer and administer SANs and pooled storage, most first-generation SAN implementers have experimented with the people and skills that are necessary. Therefore, in order to test, certify, design, install, and integrate the components, the life cycle of an advanced storage architecture project will call for a variety of teaming arrangements. Initial team arrangements used for proof-of-concept and SAN implementation programs may or may not evolve into the longer-term storage team. Thus, the organization's final determination of the storage team may be influenced by the internal lessons learned from the inception of a SAN/NAS project.
Core Storage Team and Skills
There should be a broad range of IT operational skills represented in the storage team, both in the core or nucleus team and in the ancillary team members who will participate in the SAN management role. Candidates for the core team could come from among network managers, database administration (DBA) teams, Windows NT systems administrators, Multiple Virtual Storage (MVS) operations staff, or MVS storage managers. A variety of skills, disciplines, and best-practice knowledge will be essential for the initial formation of the team. Many organizations use the creation of a SAN team as a reward to retain key personnel who are looking for more technical challenges in their career.
The value and importance of the storage pool is amplified with storage and data centralization. Since storage can now be accessed by various servers in many parts of the world, operational coverage will most likely change. For global enterprises, traditional "9-5" staff control and operations is not sufficient. Essential for 24/7 monitoring and administration, is the consideration for automation and SAN tools that are integrated with the central operation.
Now, let's look at the steady-state storage organization. This organization level occurs after the advanced storage system has been tested, certified, and installed, and the basic integration functions have been completed.
Storage Team Roles and Responsibilities
Most end users rely on the storage vendor or systems integrator to get the environment up and running and operating in a relatively stable manner. It is at this point that a storage team needs to be ready for the long-term operation and control of the storage architectures.
As a whole, the exact fit and application depends on the maturity and flexibility of the enterprise and IT department. When assembling a new storage team, these functional areas can be considered as segments to pick and choose from.
Within the enterprise, good practices and operational procedures may already be in place. These good practices and operational procedures are similar to those often found in mainframe groups.
Many successful SAN implementation teams have adopted formal processes and procedures from IT network management groups. Best practices do not need to be started from scratch for the storage team. The best practices must be borrowed and modified from other mature IT areas within the enterprise.
Network Engineering and SAN/Storage Implementation
Many enterprises have been faced with the choice of turning the SAN fabric implementation over to the existing network engineering or network administration team. There are a few cases when traditional network administration should be considered:
- The IP network segments of the LAN/WAN infrastructure should be managed by existing network management teams when a NAS or IP storage architecture is being considered for pooled storage.
- The IP stack on the host server, network interface cards (NIC), Ethernet cabling, routers, hubs, and bridges should all be managed by the IP network team(s) for span and control.
- Only the NAS appliance or NFS/CIFS servers and associated storage systems should be administered by a separate storage team.
- The network group will often be responsible for the DWDM or ATM converters involved with the asynchronous segment of the SAN extension when storage wide area networks (SWANs) are being considered in the storage topology.
Except for the cases previously mentioned, most organizations determine that FC based topologies and the associated storage arrays will be administered and managed by a net-new group within the IT organization. Network engineering has not, in the past, been accountable for SCSI connections to disk. With that in mind, a fundamental change from SCSI to FC does not provide justifiable reason to give this segment to the network group.
Furthermore, closely knit functions that may be applied with host software (HBA persistent binding, switch zoning, or LUN allocation mapping in the storage array) are the administrative functions of LUN security, zoning, and volume management. Also, end-to-end support will be difficult to trace and may cause accountability and coordination nightmares, if the chain of control is segmented with various groups or organizations that are responsible for different segments.
The network group must also be willing to take control of up-stream (HBA, world wide names (WWN), and software agents) and downstream (storage management) functions that go with the SAN, if they want control of the FC fabric switches and hardware. These are indivisible functions that need to be fully viewed and controlled by a single entity. Also, enterprises with best practices in SAN implementation and organization should assign the SAN fabric and storage management to a "net-new team" that is separate, but closely connected to servers, applications, and network teams.
SAN Implementation Team Success Metrics
The burden of proof for "expected" SAN or NAS features may often lie within the organization, as well as the technology itself, as a new organization is created to support a new technology. There are several compelling reasons to justify or promote pooled storage architectures. Some of the more prominent ones are listed below:
- Asset utilization improvement.
- Backup/restore improvements, reducing backup window times.
- Data valued and protected through data replication, mirroring.
- Disaster recovery improvements.
- Higher availability of data.
- Manageability, improved total cost of ownership (TCO).
- Meeting SAN and storage projects on time and on budget.
- Performance improvements for FC storage I/O.
- Reduced headcount per terabyte (TB) of storage, improved server-to-administrator ratios.
- Space reclamation and data center consolidation.
- Storage growth management and scalability.
Server Consolidation, Storage Consolidation
Reduction of Windows NT or UNIX servers through consolidation or pooled storage should see the system administration (SA) staff yield a measurable drop over some time period. Since the data management workload will be transferred from the SAs to storage administrators, it is theorized that with the separation of servers and disk, a higher ratio of SAs-per-server can be realized. Before and after the SAN implementation, time-based server inventory and the associated staff levels could calculate the full-time equivalent (FTE) system administrators.
Managed Storage Ratios
Pooled storage increases the total capacity of data management duties that can be assigned to a database administrator (DBA) or storage administrator. When compared to data growth by server, implementation of central, pooled, or SAN strategies can reduce the staff growth requirements.
SLA/OLA Qualitative Metrics
For IT departments that operate with service level agreement SLA or operating level agreements (OLA), the compliance to these cases could be traced to the storage team (for those elements that relate to storage). Examples might be:
- Business hours lost per year, correlated to the storage team.
- Cost of billed $/GB/month.
- Mean time to repair (MTTR), mean time between failures (MTBF) of components and design.
- Meeting backup and restore windows.
- Successful disaster recovery (DR) tests.
- The storage management team's impact on new business initiatives.
Interdependencies with Other IT Organizations
Finally, specialized storage teams, such as those that administer pooled storage systems, will have dependencies on other existing organizations within the IT department. As previously mentioned, the formation of a specialized team may be an evolutionary activity. So, the close inclusion of other disciplines in the first few months is essential. As the storage team improves its role and becomes more efficient, these associations will have some long-term implications. These dependencies will also be built into the team, in the case where the storage team is a virtual team (that is, people are loaned or assigned to participate in the storage functions). A tour-of-duty rotation scheme will allow for a variety of opportunities to learn and refresh skills, and can provide improvements in the application of skills for a longer time.
Summary and Conclusions
Many enterprises will be required to make changes to their IT organizations to effectively manage the new technology, especially with the rapid acceptance of pooled storage architectures, storage area networks (SANs), and consolidated data management. Some of these changes may be subtle. But, with SAN implementation, a new set of skills, assignments, roles, and responsibilities must be carefully considered.
Decisions on SAN implementation topology will have varying impacts on the organizational options that must be considered. Most IT shops already have in place best practices and procedures needed by the new SAN implementation team, but these practices and procedures must be assembled and slightly modified for this new technology. Depending on the phase of SAN implementation, requisite internal skills will need to be purchased, borrowed, developed, and recruited. Therefore, in order to document effectiveness of the SAN and storage operations, there are meaningful measurements to apply to the SAN implementation team.
Organizational needs should not be left to chance, or left out altogether as part of the planning and design of the SAN, especially when considering adaptation of pooled storage and SAN implementation technology. The organization's strengths, weaknesses, and adaptability will impact SAN implementation. So, these options need to be carefully reviewed in parallel with all technical decisions under consideration.
Implementation planning for the organization and staff for a medium-to-large SAN project should not be an afterthought. This technology has as much impact to some staff and reporting structures as direct attached storage has to servers. The enterprise must also consider in the early planning and architectural stages, the separation of storage from general enterprise computing and the storage management staff skills needed to manage company data.
Finally, so many of the recommendations and options presented in this article, should be evaluated and implemented only when deemed appropriate to the enterprise, since no two companies are the same. IT departments that proactively plan for the SAN team in conjunction with the SAN implementation, tend to sort out the problems of span and control early in the deployment, and have more successful technology transitions. Best practices and documented procedures for storage management, control, and operation, usually outweigh benefits of any software or automation product. Through the use of pooled storage systems, clear objectives, operational metrics, and success criteria, can empower the team to achieve the greatest results.
With that in mind, Part II of the Basics of SAN Implementation discusses the implementation of SANs with regards to backups, clusters, appliances and database applications. See you there!
About the Author :John Vacca is an information technology consultant and author. Since 1982, John has authored 36 technical books including The Essential Guide To Storage Area Networks, published by Prentice Hall. John was the computer security official for NASA's space station program (Freedom) and the International Space Station Program, from 1988 until his early retirement from NASA in 1995. John can be reached at firstname.lastname@example.org.