The New York Independent System Operator (NYISO) is responsible not only for managing New York’s electricity transmission grid — a 10,775 mile network of high-voltage lines that carry electricity throughout the state — but also for maintaining a highly reliable and available IT infrastructure too. Since its inception in 1999, NYISO has partnered with Veritas (now Symantec), among others, to help the organization do just that.
Most recently, NYISO turned to the Veritas Storage Foundation and Cluster Server by Symantec to ensure the availability of its critical information systems, and to achieve its goal of tying together clustering services and server virtualization.
The Storage Foundation 5.0 products give enterprise users visibility and control over complex data center storage environments. The Veritas Cluster Server 5.0 helps organizations reduce planned and unplanned application and database downtime. Both products are core components of the vendor’s Data Center Foundation solution family.
NYISO, which provides electricity to almost 20 million New Yorkers, has now standardized on Symantec software and services for data center automation, storage management and data protection, in the process improving management and efficiency and lowering costs.
Maintaining and operating five data centers in upstate New York was taking up too many resources at NYISO. The company’s IT team knew there was a better way, so in 2006, the organization set out to reduce the number of data centers by three. NYISO has approximately 425 employees, about 120 of whom work in the IT department.
“We wanted to consolidate data center space as well as achieve greater availability of our customer database services,” said Mike DeSocio, supervisor of the services and storage group at NYISO in Greenbush, N.Y.
At the time, the organization had more than 450 standalone Unix servers — in addition to other servers running other operating systems — running many Oracle databases that supported the company’s internal applications, such as reporting to internal groups, as well as external customer applications, such as bidding and financial services to market participants.
NYISO was also running a variety of replication and availability software from vendors such as Oracle, BMC and Hitachi (for the company’s SAN).
“We wanted to minimize the number of services, standardize on replication and standardize on the look and feel of our switchover capability from four to five flavors of failover for database services to one,” said DeSocio.
Standardization at NYISO would help achieve a high level of local system availability as well as high availability at the company’s disaster recovery site, officials reasoned.
The organization also ran servers for testing and application development. Because NYISO operates a round-the-clock business, it performs critical functions for full-time monitoring of its electricity infrastructure and also runs an equivalent IT disaster recovery site. The organization worried about configuration mishaps in its production environment from its quality assurance (QA) and development servers.
Thus, another objective at NYISO was to consolidate its QA and development servers through virtualization as a way to improve utilization and maintain high availability of the prime data center. NYISO’s initiative was to tie clustering services to server virtualization.
In the fall of 2006, the organization created a requirements document with criteria that included ease of use and ease of set-up and maintenance. On the technology side, a solution had to be compatible with the company’s Hitachi SAN and also support Oracle 10g.
The document was forward to several vendors, including professional services provider and VAR ServerWare Corp. of Pittsford, N.Y, a Symantec partner that suggested the Veritas products. After listening to presentations from three vendors, NYISO ultimately went with a solution that included Solaris, Oracle and Veritas Cluster Server.
According to DeSocio, NYISO made its decision based on the fact that the employyes were already skilled on the Solaris OS, Oracle DBMS and clustering with Veritas. Several years ago, the company put up a couple of standalone clusters using Veritas 3.5.
Discussions about the IT architecture included in-house IT experts as well as consultants from ServerWare, who provided high-level expertise on the Veritas Cluster Server. Having worked with NYISO in the past, the VAR was familiar with the organization’s IT environment.
By last January, NYISO had put together a working pilot. The pilot included the primary data center and one of five of the alternate data centers. “We split a cluster with zero downtime and migrated to a new facility,” DeSocio explained. A cluster consists of four Sun boxes, with two located at the primary data center and two at an alternate data center.
“The new architecture gave us the ability to run an Oracle database on one of four different servers, and it also reduces the overhead of having the OS running on boxes not needed for production,” said DeSocio.
Doing More With Less
By February 2007, NYISO consolidated to two data centers. Failover, which used to take three disciplines (SAN, file system servers and Oracle DBMS) of administration on a conference call, is now executed at the push of a button.
NYISO will soon add two more nodes to its cluster, one at the primary data center and another at the disaster recovery site.
The company runs a Hitachi SAN with 10 Hitachi frames, for a capacity of 400 terabytes. IT also uses Hitachi’s TrueCopy heterogeneous remote replication software and ShadowImage in-system replication software. NYISO is also using the Veritas Storage Foundation agent for Hitachi to monitor replication status and reverse replication roles upon site failover.
The Hitachi SAN is managed by a Brocade director switch. The backup part of the Hitachi SAN runs Veritas NetBackup on several IBM servers, and the organization’s Sun storage technology includes tape drives and tape silos.
While NYISO still has much work to do, Reginald Harnish, manager of infrastructure services at NYISO, acknowledged the many benefits provided by the Symantec solutions, including increased utility around future costs, improved time to market and the ability to standardize on storage management. “Consolidation has also allowed us to reduce resources spent on space and cooling,” he said.
The organization reduced the number of Unix servers by about 100. The real savings, however, is the number of servers the company won’t be purchasing as it moves forward, as new server purchases have dropped from 30 to 40 a year to the mid-single digits.