Disaster Recovery and Continuity for the Database Administrator, Part 2
As we noted in Part 1 of this series, database administrators are critical to the success of any disaster recovery scenario.
There are many key roles that are critical to the success of the database administrator. A server administrator will have to install and set up the server. A system administrator will be needed to install and set up the operating system. A storage administrator will be necessary to duplicate the disks accordingly. Application developers will need to assist with troubleshooting errors detected by the user community. These are some of the people that a database administrator will rely on.
Many, if not all, of these steps can be accomplished prior to any disaster and tested. There can also be problems at the time of failover where some of these areas may need to be revisited. The database administrator may know who to call and work with during normal times, but what happens when a disaster strikes and some primary support personnel are not available? They could be taking care of injured family members or injured themselves. What if your database administrator is not available? Contingencies for these scenarios should be put in place.
It is imperative for employees to know whom to call when they have an issue.
One of the best ways to avoid a situation with availability is cross-training employees. An employee that knows more than one job function can become essential and can play a key role during a disruption by knowing more than one area or job function.
Some people may not be able to make it to the recovery site, leaving some areas not covered, noted Eric Maiwald and William Sieglein in "Security Planning & Disaster Recovery" (McGraw-Hill). The cross-training should not be a complete shift from their normal profession, unless requested by the employee. What is usually better is to have an employee learn a skill that is new, but in the same profession they are currently engaged.
For example, Oracle database administrators can cross-train as SQL Server database administrators. They are already familiar with the concepts, SQL, structures and other features of database administration. It should mostly be a matter of learning the different toolsets for the new database software. This can be a win-win for the employee and the organization.
The employee learns a valuable new skill that can enhance their career. The organization gains an employee that has multiple skill sets that can be called upon in times of normalcy and times of crisis.
Requirements for a database will drive the type of backups you make for it. If a database can have several hours of downtime and the last night backup will work sufficiently, then a full backup will be fine. If little to no downtime or little to no data loss is acceptable, then full backups will not do the job.
Technologies such as remote mirroring will have to be investigated. In remote mirroring, all changes made to the production system are copied to the disaster recovery site. This is normally considered in an asynchronous context, since most disaster recovery sites are at some distance away from the primary site. When a fail over is called for, databases can be recovered with the mirrored data for business continuance.
Data replication is another technology that can keep disaster recovery databases updated. The native settings of the software replicate changes as they occur from production databases to databases at the disaster recovery site. This can be altered so that changes are applied on a schedule, like every four hours. This would be for a data recovery scenario in case a user made an error. The database administrator could use the data from the disaster recovery database to correct the error in production because the changes had been delayed.
Installation of database software should be a fairly routine task for a database administrator. It should also be the same across servers with the same database versions. Installation and setup should be well documented. There is always the possibility that a database administrator will not be available when a fail over is called for. Clear and concise, step by step directions will give technical professionals from another area the ability to stand in for a missing database administrator and set up the database software.
This being said, each production server is different. Certain things may need to be done to prepare the database. Special scripts will sometimes need to run, or jobs to load or unload data. These steps for individual databases and the order in which they should execute also need to be well documented.