By W. Curtis Preston Chapter 4 SAN Backup and Recovery In this chapter: Overview LAN-Free Backups Client-Free Backups Server-Free Backups LAN-Free, Client-Free, or Server-Free? Overview A SAN typically consists of multiple servers, online storage (disk), and offline storage (tape or optical), all of which are connected to a Fibre Channel switch or hub–usually a switch. […]
By W. Curtis Preston
In this chapter:
Overview
LAN-Free Backups
Client-Free Backups
Server-Free Backups
LAN-Free, Client-Free, or Server-Free?
A SAN typically consists of multiple servers, online storage (disk), and offline storage (tape or optical), all of which are connected to a Fibre Channel switch or hub–usually a switch. You can see all these SAN elements in Figure 4-1. Once the three servers in Figure 4-1 are connected to the SAN, each server in the SAN can be granted full read/write access to any disk or tape drive within the SAN. This allows for LAN-free, client-free, and server-free backups, each represented by a different-numbered arrow in Figure 4-1.
TIP:
Pretty much anything said in this chapter about Fibre Channel-based SANs will soon be true about iSCSI-based SANs. To visualize how an iSCSI SAN fits into the pictures and explanations in this chapter, simply replace the Fibre Channel HBAs with iSCSI-capable NICs, and the Fibre Channel switches and hubs with Ethernet switches and hubs, and you’ve got yourself an iSCSI-based SAN that works essentially like the Fibre Channel-based SANs discussed in this chapter. The difficulty will be in getting the storage array, library, and software vendors to support it.
To my knowledge, I am the first to use the term client-free backups. Some backup software vendors refer to this as server-free backups, and others simply refer to it by their product name for it. I felt that a generic term was needed, and I believe that this helps distinguish this type of backups from server-free backups, which are defined next.
No backup is completely LAN-free, client-free, or server-free. The backup server is always communicating with the backup client in some way, even if it’s just to get the metadata about the backup. What these terms are meant to illustrate is that the bulk of the data is transferred without using the LAN (LAN-free), the backup client (client-free), or the backup server (server-free).
Client-Free or Server-Free?
I am well aware that some vendors refer to client-free backups as server-free backups. I find their designation confusing, and I’m hoping that my term client-free backups will catch on. The reason that they refer to client-free backups as server-free is that the backup world turns things around. What you refer to as a server, the backup system refers to as a client. Although you consider it a server, it’s a client of the backup and recovery system. When it comes to the backup and recovery system, the server is the system that is controlling all of the backups. I am trying to keep things consistent by referring to backups that don’t go through a client (but do go through the backup server) as client-free backups. In order for me to call something server-free, the data path must not include any server of any kind. The only backup like that is an extended copy backup as described in the previous section. Therefore, I refer only to extended copy backups as server-free backups. That’s my story and I’m sticking to it.
Authors
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
As discussed in Chapter 1, LAN-free backups allow you to use a SAN to share one of the most expensive components of your backup and recovery system–your tape or optical library and the drives within it. Figure 4-2 shows how this is simply the latest evolution of centralized backups. There was a time when most backups were done to locally attached tape drives. This method worked fine when data centers were small, and each server could fit on a single tape. Once the management of dozens (or even hundreds) of individual tapes became too much or when servers would no longer fit on a tape, data centers started using backup software that allowed them to use a central backup server and back up their servers across the LAN. (The servers are now referred to by the backup system as clients.)
This methodology works great as long as you have a LAN that can support the amount of network traffic such backups generate. Even if you have a state-of-the-art LAN, you may find individual backup clients that are too big to back up across the LAN. Also, increasingly large amounts of system resources are required on the backup server and clients to back up large amounts of data across the LAN. Luckily, backup software companies saw this coming and added support for remote devices. This meant that you could again decentralize your backups by placing tape drives on each backup client. Each client would then be told when and what to back up by the central backup server, but the data would be transferred to a locally attached tape drive. Most major software vendors also allowed this to be done within a tape library. As depicted in Figure 4-2, you can connect one or more tape drives from a tape library to each backup client that needs them. The physical movement of the media within the library is then managed centrally–usually by the backup server.
Although the backup data at the bottom of Figure 4-2 isn’t going across the LAN, this isn’t typically referred to as LAN-free backups. The configuration depicted at the bottom of Figure 4-2 is normally referred to as library sharing, since the library is being shared, but the drives aren’t. When people talk about LAN-free backups, they are typically referring to drive sharing, where multiple hosts have shared access to an individual tape drive. The problem with library sharing is that each tape drive is dedicated to the backup client to which it’s connected.[2] The result is that the tape drives in a shared library go unused most of the time.
As an example, assume we have three large servers, each with 1.5 TB of data. Five percent of this data changes daily, resulting in 75 GB of incremental backups per day per host.[3] All backups must be completed within an eight-hour window, and the entire host must be backed up within that window, requiring an aggregate transfer rate of 54 MB/s[4] for full backups. If you assume each tape drive is capable of 15 MB/s,[5] each host needs four tape drives to complete its full backup in one night. Therefore, we need to connect four tape drives to each server, resulting in a configuration that looks like the one at the bottom of Figure 4-2. While this configuration allows the servers to complete their full backups within the backup window, that many tape drives will allow them to complete the incremental backup (75 GB) in just 20 minutes!
An eight-hour backup window each night results in 240 possible hours during a month when backups can be performed. However, with a monthly full backup that takes eight hours, and a nightly incremental backup that takes 20 minutes, they are going unused for 228 out of their 240 available hours.
However, suppose we take these same three servers and connect them and the tape library to a SAN as illustrated in Figure 4-3. When we do so, the numbers change drastically. As you can see in Figure 4-3, we’ll connect each host to a switched fabric SAN via a single 100-MB Fibre Channel connection. Next, we must connect the tape drives to the SAN. (For reasons explained later, however, we will only need five tape drives to meet the requirements discussed earlier.) If the tape drives supported Fibre Channel natively, we can just connect them to the SAN via the switch. If the tape drives can connect only via standard parallel SCSI cables, we can connect the five tape drives to the SAN via a Fibre Channel router, as can be seen in Figure 4-3. The routers that are shipping as of this writing can connect one to six SCSI buses to one or two Fibre Channel connections. For this example, we will use a four-to-one model allowing us to connect our five tape drives to the SAN via a single Fibre Channel connection. (As is common in such configurations, two of the drives will be sharing one of the SCSI buses.)
With this configuration, we can easily back up the three hosts discussed previously. The reason that we only need only five tape drives is that in the configuration shown in Figure 4-3, we can use dynamic drive sharing software provided by any of the major backup software companies. This software performs the job of dynamically assigning tape drives to the hosts that need them. When it’s time for a given host’s full backup, it is assigned four of the five drives. The hosts that aren’t performing a full backup will share the fifth tape drive. By sharing the tape drives in this manner, we could actually backup 25 hosts this size. Let’s take a look at some math to see how this is possible.
Assume we have 25 hosts, each with 1.5 TB of data. (Of course, all 25 hosts need to be connected to the SAN.) With four of the five tape drives, we can perform an eight-hour full backup of a different host each night, resulting in a full backup for each server once a month. With the fifth drive, we can perform a 20-minute incremental backup of the 24 other hosts within the same eight-hour window.[6] If this was done with directly connected SCSI, it would require connecting four drives to each host, for a total of 100 tape drives! This, of course, would require a really huge library. In fact, as of this writing, there are few libraries available that can hold that many tape drives. Table 4-1 illustrates the difference in the price of these two solutions.
|
|
Parallel SCSI solution |
SAN solution |
||
|---|---|---|---|---|
|
|
Quantity |
Total |
Quantity |
Total |
|
Tape drives |
100 |
$900K |
5 |
$45K |
|
HBAs |
25 |
$25K |
25 |
$50K |
|
Switch |
None |
|
2 |
$70K |
|
Router |
None |
|
1 |
$7K |
|
Device server software |
Same |
|
Same |
|
|
Tape library |
Same |
|
Same |
|
|
Total |
|
$925K |
|
$172K |
TIP:
Whether or not a monthly full backup makes sense for your environment is up to you. Unless circumstances require otherwise, I personally prefer a monthly full backup, nightly incremental backups, followed by weekly cumulative incremental backups.
Authors
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
Each solution requires some sort of HBA on each client. Since Fibre Channel HBAs are more expensive, we will use a price of $2,000 for the Fibre Channel HBAs, and $1,000 for the SCSI HBAs.[7] We will even assume you can obtain SCSI HBAs that have two SCSI buses per card. The SCSI solution requires 100 tape drives, and the SAN solution requires five tape drives. (We will use UHrium LTO drives for this example, which have a native transfer speed of 15 MB/s and a discounted price of $9,000.) Since the library needs to store at least one copy of each full backup and 30 copies of each incremental backup from 25 1.5-TB hosts, this is probably going to be a really large library. To keep things as simple as we can, let’s assume the tape library for both solutions is the same size and the same cost. (In reality, the SCSI library would have to be much larger and more expensive.) For the SAN solution, we need to buy two 16-port switches, and one four-to-one SAN router. Each solution requires purchasing device server licenses from your backup software vendor for each of the 25 hosts, so this will cost the same for each solution. (The SAN solution licenses might cost slightly more than the SCSI solution licenses.) As you can see in Table 4-1, there is a $700,000 difference in the cost of these two solutions.
What about the original requirements above, you ask? What if you only need to back up three large hosts? A three-host SAN solution requires the purchase of a much smaller switch and only one router. It requires 12 tape drives for the SCSI solution but only five for the SAN solution. As you can see in Table 4-2, even with these numbers, the SAN solution is over $40,000 cheaper than the SCSI solution.
|
|
Parallel SCSI solution |
SAN solution |
||
|---|---|---|---|---|
|
|
Quantity |
Total |
Quantity |
Total |
|
Tape drives |
12 |
$108K |
5 |
$45K |
|
HBAs |
3 |
$3K |
3 |
$6K |
|
Switch |
None |
|
1 |
$8K |
|
Router |
None |
|
1 |
$7K |
|
Device server software |
Same |
|
Same |
|
|
Tape library |
Same |
|
Same |
|
|
Total |
|
$111K |
|
$66K |
The data in the previous section shows that sharing tape drives between servers can be a Good Thing. But how can multiple servers share the same physical tape drive? This is accomplished in one of two ways. Either the vendor uses the SCSI reserve and release commands, or they implement their own queuing system.
Tape drives aren’t the first peripherals that needed to be shared between two or more computers. In fact, many pre-SAN high availability systems were based on disk drives that were connected to multiple systems via standard, parallel SCSI. (This is referred to as a multiple-initiator system.) In order to make this possible, the commands reserve and release were added to the SCSI-2 specification. The following is a description of how they were intended to work.
Each system that wants to access a particular device issues the SCSI reserve command. If no other device has previously reserved the device, the system is granted access to the device. If another system has already reserved the device, the application requesting the device is given a resolution conflict message. Once a reservation is granted, it remains valid until one of the following things happen:
If you combine the idea of the SCSI reserve command with a SCSI bus that is connected to more than one initiator (host HBA), you get a configuration where multiple hosts can share the same device by simply reserving it prior to using it. This device can be a disk drive, tape drive, or SCSI-controlled robotic arm of a tape library. The following is an example of how the configuration works.
Each host that needs to put a tape into a drive attempts to reserve the use of the robotic arm with the SCSI reserve command. If it’s given a resolution conflict message, it waits a predetermined period of time and tries again until it successfully reserves it. If it’s granted a reservation to the robotic arm, it then attempts to reserve a tape drive using the SCSI reserve command. If it’s given a resolution conflict message while attempting to reserve the first drive in the library, it continues trying to reserve each drive until it either successfully reserves one or is given a resolution conflict message for each one. If this happens, it issues a SCSI release command for the robotic arm and then waits a predetermined period of time and tries again. It continues to do this until it successfully puts a tape into a reserved drive. Once done, it can then back up to the drive via the SAN.
This description is how the designers intended it to work. However, shared drive systems that use the SCSI reserve/release method run into a number of issues, so many people consider the SCSI reserve/release command set to be fundamentally flawed. For example, one such flaw is that a SCSI bus reset releases the reservation. Since SCSI bus resets happen under a number of conditions, including system reboots, it’s highly possible to completely confuse your reservation system with the reboot of only one server connected to the shared device. Another issue with the SCSI reserve/release method is that not all platforms support it. Therefore, a shared drive system that uses the SCSI reserve/release method is limited to the platforms on which these commands are supported.
Most backup software vendors use the third-party queuing method for shared devices. When I think of how this works, I think my two small daughters. Their method of toy sharing is a lot like the SCSI reserve/release method. Whoever gets the toy first gets it as long as she wants it. However, as soon as one of them has one toy, the other one wants it. The one who wants the toy continually asks the other one for it. However, the one that already has the toy issues a resolution conflict message. (It sounds like, “Mine!”) The one that wants the toy is usually persistent enough, however, that Mom or Dad have to intervene. Not very elegant, is it?
A third-party queuing system is like having Daddy sit between the children and the toys, where each child has full knowledge of what toys are available, but the only way they can get a toy is to ask Daddy for it. If the toy is in use, Daddy simply says, “You can’t have that toy right now. Choose another.” Then once the child asks for a toy that’s not being used, Daddy hands it over. When the child is done with the toy, Daddy returns it to the toy bin.
There are two main differences between the SCSI reserve/release method and the third-party queuing method:
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
In addition to accomplishing drive sharing via different methods, backup software companies also differ in how they allow libraries and their drives to be shared. In order to understand what I mean, I must explain a few terms. A main server is the central backup server in any backup configuration. It contains the schedules and indexes for all backups and acts as a central point of control for a group of backup clients. A device server has only tape drives that receive backup data from clients. The main server controls its actions. This is known as a three-tiered backup system, where the main server can tell a backup client to back up to the main server’s tape drives or to the tape drives on one of its device servers. Some backup software products don’t support device servers and require all backups to be transmitted across the LAN to the main server. This type of product is known as a two-tiered backup system. (A single-tiered backup product would support only backups to a server’s own tape drives. An example of such a product would be the software that comes with an inexpensive tape drive.)
One of the truly beautiful things about sharing tape drives in a library via a SAN is what happens when it’s time to restore. First, consider the parallel study illustrated in Table 4-2. Since only three drives are available to each host, only three drives will be available during the restores as well. Even if your backup software had the ability to read more than three tapes at a time, you wouldn’t be able to do so.
However, if you are sharing the drives dynamically, a restore could be given access to all available drives. If your backups occur at night, and most of your restores occur during the day, you would probably be given access to all drives in the library during most restores. Depending on the capabilities of your backup software package, this can drastically increase the speed of your restore.
TIP:
Few backup software products can use more drives during a restore than they did during a backup, but that isn’t to say that such products don’t exist.
As of this writing, there are at least three other ways to share devices without using a SAN. Although this is a chapter about SAN technology, I thought it appropriate to mention them here.
The Network Data Management Protocol (NDMP), originally designed by Network Appliance and PDC/Intelliguard (acquired by Legato) to back up filers, offers another way to share a tape library. There are a few vendors that have tape libraries that can be shared via NDMP.
Here’s how it works. First, the vendors designed a small computer system to support the “tape server” functionality of NDMP:
They then connect the SCSI bus of this computer to a tape drive in the library, and the Ethernet port of this computer to your LAN. (Current implementations have one of these computers for each tape drive, giving each tape drive its own connection to the LAN.) The function of the computer system is to convert data coming across the LAN in raw TCP/IP format and to write it as SCSI data via a local device driver which understands the drive type. Early implementations are using DLT 8000s and 100-Mb Ethernet, because the speed of a 100-Mb network is well matched to a 10-MB/s DLT. By the time you read this, there will probably be newer implementations that use Gigabit Ethernet and faster tape drives.
Not only does this give you a way to back up your filers; it also offers another option for backing up your distributed machines. As will be discussed in Chapter 7, NDMP v4 supports backing up a standard (non-NDMP) backup client to an NDMP-capable device. Since this NDMP library looks just like another filer to the backup server, you can use it to perform backups of non-NDMP clients with this feature. Since the tape library does the job of tracking which drives are in use and won’t allow more than one backup server to use it at a time, you can effectively share this tape library with any backup server on the LAN.
SCSI over IP, also known as iSCSI, is a rather new concept when compared to Fibre Channel, but it’s gaining a lot of ground. A SCSI device with an iSCSI adapter is accessible to any host on the LAN that also has an iSCSI interface card. Proponents of iSCSI say that Fibre Channel-based SANs are complex, expensive, and built on standards that are still being finalized. They believe that SANs based on iSCSI will solve these problems. This is because Ethernet is an old standard that has evolved slowly in such a way that almost all equipment remains interoperable. SCSI is also another standard that’s been around a long time and is a very established protocol.
The one limitation to iSCSI today is that it’s easy for a host to saturate a 100-MB/s Fibre Channel pipe, but it’s difficult for the same host to saturate a 100-MB/s (1000 Mb or 1 Gb) Ethernet pipe. Therefore, although the architectures offer relatively the same speed, they aren’t necessarily equivalent. iSCSI vendors realize this and are therefore have developed network interface cards (NICs) that can communicate “at line speed.” One way to do this is to offload the TCP/IP processing from the host and perform it on the NIC itself. Now that vendors have started shipping gigabit NICs that can communicate at line speed, iSCSI stands to make significant inroads into the SAN marketplace. More information about iSCSI can be found in Appendix A.
There are some vendors that provide shared access to SCSI devices via standard parallel SCSI. Since all the LAN-free implementations that I have seen have used Fibre Channel, it’s unclear how well such products would be supported by drive-sharing products.
Problems Caused by Reboots
I have heard reports of difficulties caused by an individual host rebooting while a drive on the SAN is in use by another host. On some operating systems, busy drives aren’t seen by the operating system, and thus aren’t available to use. Even worse, it sometimes causes the available drives’ names to be changed, resulting in a completely inoperable set of drives. Some vendors have worked around the missing-drives problems using intelligent SAN routers. Some vendors have worked around name-change problems by tying the virtual name of the drive to the drive’s actual serial number. Make sure you talk to your SAN hardware and software vendors about how they handle this situation.
LAN-free backups solve a lot of problems by removing backup traffic from the LAN. When combined with commercial interfaces to database backup APIs, they can provide a fast way to back up your databases. However, what happens if you have an application that is constantly changing data, and for which there is no API? Similarly, what if you can’t afford to purchase the interface to that API from your backup software company? Usually, the only answer is to shut down the application during the entire time of the backup.
One solution to this problem is to use a client-free backup solution, which is covered in the next section. However, client-free backups are expensive, and they are meant to solve more problems (and provide more features) than this. If you don’t need all the functionality provided by client-free backups, but you still have the problem described earlier, perhaps what you need is a snapshot. In my book Unix Backup and Recovery, I introduced the concept of snapshots as a feature I’d like to see integrated into commercial backup and recovery products, and a number of them now offer it.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
A snapshot is a virtual copy of a device or filesystem. Think of it as a Windows shortcut or Unix symbolic link to a device that has been frozen in time. Just as a symbolic link or shortcut isn’t really a copy of a file or device that it points to, a snapshot is a symbolic (or virtual) representation of a file or device. The only difference between a snapshot and a shortcut or symbolic link is that the snapshot always mimics the way that file or device looked at the time the snapshot was taken.
In order to take snapshots, you must have snapshot software. This software can be found in any of the following places:
Let’s review why we are looking at snapshots. Suppose you perform LAN-free backups but have an application for which there is no API or can’t afford the API for an application. You are therefore required to shut down the application during backups. You want something better but aren’t yet ready for the cost of client-free backups, nor do you need all the functionality they provide. Therefore, you need the type of snapshots that are available only from an advanced filesystem, a host-based volume manager, or a backup software add-on product that emulates this functionality. An enterprise storage array that can create snapshots (that are visible from the host of which the snapshot was taken) also works fine in this situation, but it isn’t necessary. Choose whichever solution works best in your environment. Regardless of how you take the snapshot, most snapshot software works essentially the same way.
When you create a snapshot, the snapshot software records the time at which the snapshot was taken. Once the snapshot is taken, it gives you and your backup utility another name through which you may view the snapshot of the device or filesystem. It looks like any other device or filesystem, but it’s really a symbolic representation of the device. Creating the snapshot doesn’t actually copy data from diska to diska.snapshot, but it appears as if that’s exactly what happened. If you look at diska.snapshot, you’ll see diska exactly as it looked at the moment diska.snapshot was created.
Creating the snapshot takes only a few seconds. Sometimes people have a hard time grasping how the software can create a separate view of the device without copying it. This is why it’s called a snapshot; it doesn’t actually copy the data, it merely took a “picture” of it.
Once the snapshot has been created, most snapshot software (or firmware in the array) monitors the device for activity. When it sees that a block of data is going to change, it records the “before” image of that block in a special logging area (often called the snapshot device). Even if a particular block changes several times, it only needs to record the way it looked before the first change occurred.[9]
For details on how this works, please consult your vendor’s manual. When you view the device or filesystem via the snapshot virtual device or mount point, it watches what you’re looking for. If you request a block of data that has not changed since the snapshot was taken, it retrieves that block from the original device or filesystem. However, if you request a block of data that has changed since the snapshot was taken, it retrieves that block from the snapshot device. This, of course, is completely invisible to the user or application accessing the data. The user or application simply views the device via the snapshot device or mount point, and where the blocks come from is managed by the snapshot software or firmware.
Now that you can create a “copy” of your system in just a few seconds, you have a completely different way to back up an unsupported application. Simply stop the application, create the snapshot (which only takes a few seconds) and restart the application. As far as the application is concerned, the backup takes only a few seconds. However, there is a performance hit while the data is being backed up to the locally attached tape library (that is being shared with other hosts via a LAN-free backup setup). The degree to which this affects the performance of your application will, of course, vary. The only way to back up this data without affecting the application during the actual transfer of data from disk to tape is to use client-free backups.
TIP:
One vendor uses the term “snapshot” to refer to snapshots as described here, and to describe additional mirrors created for the purpose of backup, which is discussed in the section “Client-Free Backups.” I’m not sure why they do this, and I find it confusing. In this book, the term snapshot will always refer to a virtual, copy-on-write copy as discussed here.
LAN-free backups solve a lot of problems. They allow you to share one or more tape libraries between your critical servers, allowing each to back up much faster than they could across the LAN. It removes the bulk of the data transfer from the LAN, freeing your LAN for other uses. It also reduces the CPU and memory overhead of backups on the clients, because they no longer have to transmit their backup data via TCP/IP. However, there are still downsides to LAN-free backups. Let’s take a look at a LAN-free backup system to see what these downsides are.
A typical LAN-free backup system is shown in Figure 4-4, where the database resides on disks that are visible to the data server. These disks may be a set of discreet disks inside the data server, a disk array with or without mirroring, or an enterprise storage array. In order to back up the data, the data must be read from the disks by the data server (1a) and transferred across the data server’s backplane, CPU, and memory via some kind of backup software. It’s then sent to tape drives on the data server (1b). This is true even if you use the snapshot technology discussed earlier. Even though you’ve created a static view of the data to back up, you still must back up the data locally.
This, of course, requires using the CPU, memory, and backplane of the data server quite a bit. The application is negatively affected, or even shut down, during the entire time it takes to transfer the database from disk to tape. The larger the data, the greater the impact on the data server and the users. Also, traditional database recovery (including snapshot recovery systems like those described at the end of the previous section) requires transferring the data back across the same path, slowing down a recovery that should go as fast as possible.
With this type of setup, a recovery takes as long or longer than the backup and is limited by the I/O bandwidth of the data server. This includes only the recovery of the database files themselves. If it’s a database you are recovering, the replaying of the transaction logs adds a significant amount of time on top of that.
Let’s take a look at each of these limitations in more detail.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
You’d think that backing up data to locally attached tape drives would present a minimal impact to the application. It certainly creates much less of a load than typical LAN-based backups. In reality, however, the amount of throughput required to complete the backup within an acceptable window can sometimes create quite a load on the server, robbing precious resources needed by the application the server is running. The degree to which the application is affected depends on certain factors:
Please notice that the last question asked about when the application is needed–not when it’s being used. The reason the question is worded this way is that too many businesses have gotten used to systems that aren’t available during the backup window. They have grown to expect this, so they simply don’t try to use it at that time. They would use it if they could, but they can’t–so they don’t. The question is, “Would you like to be able to access your system 24 × 7?” If the answer is yes, you need to design a backup and recovery system that creates minimal impact on the application.
Almost all applications are impacted in some way during LAN-based or LAN-free backups. File servers take longer to process file requests. If you have a database and can perform backups with the database running, your database may take longer to process queries and commits. If your database application requires you to shut down the database to perform a backup, the impact on users is much greater.
Whether you are slowing down file or database services, or you are completely halting all database activity, it will be for some period of time. The duration of this period is determined by four factors:
This is the only reason you are backing up, right? Many people fail to take recovery speed into consideration when designing a backup and recovery system when they should be doing almost the opposite. They should design a backup and recovery system in such a way that it can recover the system within an acceptable window. In almost every case, this also results in a system that can back up the system within an acceptable window.
If your backup system is based on moving data from disk to tape, and your recovery system is based on moving data from tape to disk, the recovery time is always a factor of the questions in the previous section. They boil down to two basic questions: how much data do you have to move, and what resources are available to move it?
Of course applications are affected during backups. Recovery takes as long, if not longer, than the backup. There’s simply no way to get around this, right? That was the correct answer up until just recently; however, client-free backups have changed the rules.
What if there was a way you could back up a given server’s data with almost no impact to the application? If there were any impact, it would last for only a few seconds. What if you could recover a multi-terabyte database instantaneously? Wouldn’t that be wonderful? That’s what client-free backups can do for you.
Performing client-free backup and recovery requires the coordination of several steps across at least two hosts. At one point in time, none of the popular commercial backup and recovery applications had software that could automate all the steps without requiring custom scripting by the administrator. All early implementations of client-free backups involved a significant amount of scripting on the part of the administrator, and almost all early implementations of client-free backups were on Unix servers. This was for several reasons, the first of which was demand. Many people had really large, multi-terabyte Unix servers that qualified as candidates for client-free backups. This led to a lot of cooperation between the storage array vendors and the Unix vendors, which led to commands that could run on a Unix system and accomplish the tasks required to make client-free backups possible. Since many of these tasks required steps to be coordinated on multiple computers, the rsh and ssh capabilities of Unix came in handy.
Since NT systems lacked integrated, advanced scripting support, and communications between NT machines were easy for administrators to script, it wasn’t simple to design a scripted solution for NT client-free backups. (As you will see later in this section, another key component that was missing was the ability to mount brand new drive letters from the command line.) This, combined with the fact that NT machines tended to use less storage than their monolithic Unix counterparts, meant that there were not a lot of early implementations of client-free backup on NT. However, things have changed in recent years. It isn’t uncommon to find very large Windows machines. (I personally have seen one approaching a terabyte.) Scripting and intermachine communication has improved in recent years, but the limitation of not being able to mount drives via the command line has existed until just recently.[10] Therefore, it’s good that a few commercial backup software companies are beginning to release client-free backup software that includes the Windows platform. Those of us with very large Windows machines can finally take advantage of this technology.
Windows isn’t the only platform for which commercial client-free applications are being written. As of this writing, I am aware of several products (that are either in beta or have just been released) that will provide integrated client-free backup and recovery functionality for at least four versions of Unix (AIX, HP-UX, Solaris, and Tru64).
The next section attempts to explain all the steps a client-free backup system must complete. These steps can be scripted, or they can be managed by an integrated commercial client-free backup software package. Hopefully, by reading the steps in detail, you will have a greater appreciation of the functionality client-free backups provide, as well as the complexity of the application that provide them.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
As you can see in Figure 4-5, there is SAN-connected storage that is available to at least two systems: the data server and a backup server. The storage consists of a primary disk set and a backup mirror, which is simply an additional copy of the primary disk set. (In the figure, the primary disk set is mirrored, as represented by the M1/M2.) Note that the SAN-connected storage is depicted as a single, large multihosted storage array with an internal, prewired SAN. The reason for this is that all early implementations of client-free backups have been with such storage arrays. As discussed earlier, there are now enterprise volume managers that will make it possible to do this with JBOD on a SAN, but I use the concept of a large storage array in this example because it’s the solution that’s available from the most vendors–even if it’s the most expensive.
The normal status quo of the backup mirror is that it’s left split from the primary disk set. Why it’s left split will become clear later in this section.
At the appropriate time, the backup application performs a series of tasks:
These tasks mean that the backup data is sent via the SAN to the backup server and doesn’t travel through the client at all, which is why it’s referred to as client-free backups. Again, some vendors refer to this as server-free backups, but I reserve that term for a specific type of backup that will be covered in the section, “Server-Free Backups.”
You may find yourself asking a few questions:
Before explaining this process in detail, let’s examine a few things that make client-free backups possible:
Before continuing this explanation of client-free backups, I must define some terms that will be used in the following example. Please understand that there are multiple vendors that offer this functionality, and they all use different terminology and all work slightly differently.
TIP:
The lists that follow in the rest of this chapter give examples for Compaq, EMC, and Hitachi using Exchange, Informix, Oracle, and SQL Server databases, and the Veritas Volume Manager and File System. These examples are for clarification only and aren’t meant to imply that this functionality is available only on these platforms or to indicate an endorsement of these products. These vendors are listed in alphabetical order throughout, and so no inferences should be made as to the author’s preference for one vendor over another. There are several multihosted storage array vendors and several database vendors. Consult your vendor for specifics on how this works on your platform.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
This section explains what’s involved in backing up a backup mirror to another server.
Figure 4-6 again illustrates a typical backup mirror backup configuration. There is a main backup server (Backup Server A) that is connected to a tape library (via SCSI or Fibre Channel) and connected to the data server via a LAN. The data server is connected via Fibre Channel to the storage array, and its datafiles and transaction logs reside on its primary disk sets. The datafiles have a backup mirror associated with them, and it’s connected via Fibre Channel to a device server (Backup Server B), and that device server is connected via SCSI to another tape library.
TIP:
Multihosted storage arrays have a SAN built into them. You can put extra switches or hubs between the storage array and the servers connecting to it, but it isn’t necessary unless the number of servers that you wish to connect to the storage array exceeds the number of Fibre Channel ports the array has.
To back up a database, you must back up its datafiles (and other files) and its transaction logs. The datafiles contain the actual database, and the transaction logs contain the changes to the database since the last time you backed up the datafiles. Once you restore the datafiles, the transaction logs can then replay the transactions that occurred between the time of the datafile backup and the time of the failure. Therefore, you must back up the transaction logs continually, because they are essential to a good recovery.
As shown in Figure 4-7, transaction logs are often backed up to disk (1), and then to tape (2). This is my personal preference, but it isn’t required. The reason I prefer to back up to disk and then to tape is that replaying the transaction logs is the slowest part of the restore, and having the transaction logs available on disk significantly speeds up the process. The log disk may or may not be on the storage array and doesn’t need a backup mirror.
You can also see in Figures 4-5 and 4-6 that the transaction logs don’t need to be backed up via the split mirror. The reason for this is that there is no supported methods for backing up transaction logs this way. Therefore, there is no reason to put them on the backup mirror.
As discussed previously, the backup mirror is left split from the primary disk set (3). The following list details how transaction logs are backed with Exchange, Informix, Oracle, and SQL Server:
It’s now time to back up the datafiles. In order to do that, you must:
The details of how these steps are accomplished are discussed next.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
As shown in Figure 4-8, you must establish (reattach) the backup mirror to the primary disk set (1), causing the mirror to copy to the backup mirror any regions that have been modified since the backup mirror was last established (2).
The following list shows how this works for the various products:
set mirrorset-name nopolicy
set mirrorset-name members=3
set mirrorset-name replace=disk-name
# symbcv establish -g group_name [-i]
# paircreate -g device_group
As shown in Figure 4-9, once the backup mirror is fully synchronized with the primary disk set, you have to tell the data server to put the database in backup mode (1). In most client-server backup environments, this is normally accomplished by telling the backup software to run a script prior to a backup. The problem here is the backup client, where the script is normally run, isn’t the host where the script needs to run. The client in Figure 4-9 is actually the Backup Server B, not Data Server. This means that you need to use something like rsh or ssh to pass the command from Backup Server B to Data Server. There are now ssh servers available for both Unix and Windows. The commands you need to run on Data Server will obviously depend on the application.
Here are the steps for Exchange, Informix, Oracle, and SQL Server:
net stop “Microsoft Exchange Directory” /y
net stop “Microsoft Exchange Event Service” /y
net stop “Microsoft Exchange Information Store” /y
net stop “Microsoft Exchange Internet Mail Service” /y
net stop “Microsoft Exchange Message Transfer Agent” /y
net stop “Microsoft Exchange System Attendant” /y
net stop MSSQLSERVER
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
As shown in Figure 4-9, once the database is put into backup mode, the backup mirror is split from the primary disk set (2).
This split requires a number of commands on the Compaq array. First, you set the nopolicy flag with the command set mirrorset-name nopolicy. Then you split the mirror by issuing the command reduce disk-name for each disk in the BCV. Then you create a unit name for each disk with the command add unit unit-name disk-name. Finally, you make that unit visible to the backup server by issuing the commands set unit unit-name disable_access_path=all and set unit unit-name enable_access_path=(backupserver-name).
To do this on EMC, run the command symbcv split -g device_group on the backup server. To do this on Hitachi, run the command pairsplit -g device_group. (On both EMC and Hitachi, the backup mirror devices are made visible to the backup server as part of the storage array’s configuration.)
Now that the backup mirror is split, you can take the database out of backup mode. Here are the details of taking the databases out of backup mode:
net start “Microsoft Exchange Directory” /y
net start “Microsoft Exchange Event Service” /y
net start “Microsoft Exchange Information Store” /y
net start “Microsoft Exchange Internet Mail Service” /y
net start “Microsoft Exchange Message Transfer Agent” /y
net start “Microsoft Exchange System Attendant” /y
net start MSSQLSERVER
This step is the most OS-specific and complicated step, since it involves using OS-level volume manager and filesystem commands. However, the prominence of the Veritas Volume Manager makes this a little simpler. Veritas Volume Manager is now available for HP-UX, Solaris, NT, and Windows 2000. Also, the native volume manager for Tru64 is an OEM version of Volume Manager.
On the data server, the logical volumes are mounted as filesystems (or drives) or used as raw devices on Unix for a database. As shown in Figure 4-9, you need to figure out what backup mirror devices belong to which disk group, import those disk groups to the backup server (3), activate the disk groups (which turns on the logical volumes), and mount the filesystems if there are any. This is probably the most difficult part of the procedure, but it’s possible to automate it.
It isn’t possible to assign drive letters to devices via the command line in NT and Windows 2000 without the full-priced version of the Veritas Volume Manager. Therefore, the steps in the list that follows assume you are using this product. Except where noted, the commands discussed next should work roughly the same on both Unix and NT systems. If you don’t want to pay for the full-priced volume manager product, there is an alternate, although much less automated, method discussed at the end of the Veritas-based steps.
You can write a script that discovers the names of the disk groups on the backup mirror and uses the command vxdg -n newname -t import volume-group to import the disk/volume groups from the backup server. (The -t option specifies that the new name is only temporary.) The following list is a brief summary of how this works. It’s not meant to be an exact step-by-step guide; it gives you an overall picture of what’s involved in discovering and mounting the backup mirror.
You now have a list of disk groups that can be imported from the backup mirror.
TIP:
It’s wise to perform an fsck or chkdisk of the backup mirror’s filesystems at this point.
As mentioned before, if you are running Windows and don’t wish to pay for the full-priced version of Volume Manager, you can’t assign the drive letters via the command line. Since it isn’t reasonable to expect you to go to the GUI every night during backups, an alternative method is to perform the following steps manually each time there is a configuration change to the backup mirror:
TIP:
It’s wise to perform a chkdisk of the backup mirror’s disks at this point.
This is the easy part. Define a backup configuration to back up the filesystems, drives, or raw devices that were found on the backup mirror. Since the volumes are actually imported and/or mounted on the backup server, the backup server should be listed as the client for this backup configuration. As shown in Figure 4-9, the data is sent from the backup mirror to the backup server (4a), and then to tape (4b).
The transaction logs should be backed up to disk and to tape (as shown in Figure 4-7) before, during, and after this operation.
And that’s all there is to client-free backup!Click here to buy book
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
Nobody cares if you can back up–only if you can recover. How do you recover from such a convoluted setup? This is really where the beauty of having the backup mirror stay split and mounted comes in. Unless the entire storage array is destroyed, it’s available for an immediate restore. However, let’s start with the worst-case scenario: the entire storage array was destroyed and has been replaced.
As shown in Figure 4-10, the worst possible thing that can happen is if something catastrophic happens to the storage array, destroying the primary disk set, backup mirror, and transaction log backup disk. Because of the extreme amount of proactive monitoring most of the enterprise storage arrays have, this is rather unlikely to happen. In fact, with the backup mirror left split most of the time, about the only probable way for both the backup mirror and the primary disk set to be damaged is complete destruction of the physical unit, such as by fire. The first step in this recovery is that the storage array must, of course, be repaired or reinstalled.
Once the storage array is fully functional, you need to restore the backup mirror from tape as shown in Figure 4-11 (1a-b). While this is going on, the transaction log backups can also be restored from tape to disk (2a, b, and c). This allows them to be immediately available once the backup mirror is restored.
Once the recovery of the backup mirror has been completed from tape, we can move on to the next step. (Of course, if you don’t have to recover the backup mirror from tape, you can start with the next step.)
This recovery is much more likely to occur and will occur under the three following circumstances:
The process now is to recover the primary disk set from the backup mirror, replay the transaction logs, and bring the database online.
You now need to tell the storage array to copy the data from the backup mirror back to the primary disk set. Here’s how to do so:
Since the disk-based transaction log dumps were recovered in a previous step, you can now start the transaction log restore from the disk-based transaction log backups (see Figure 4-13).
Here’s an overview of how to do this for Exchange, Informix, Oracle, and SQL Server:
$ sqlplus /nolog
> connect /as sysdba
> startup mount ;
> recover database ;
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
Once the transaction log recovery is complete, the database is fully online, whether or not the backup mirror has been fully restored to the primary disk set (see Figure 4-14).
Isn’t that recovery scenario a beautiful thing? Imagine recovering a 10-TB database in under a minute. If you need the ultimate in instant recovery, it’s difficult to beat the concept of client-free backup and recovery.
Some people love to have an entire copy of their database or file server sitting there at all times just waiting for an instant restore. Others feel that this is a waste of money. All that extra disk can cost quite a bit. They like the idea of client-free backups–backing up the data through a system other than the system that is using the data. However, they don’t want to pay for another set of disks to store the backup mirror. What do they do?
The first thing some do is to share the backup mirror. Whether the backup mirror is in an enterprise storage array or a group of JBOD that is made available on the SAN for that purpose, it’s possible to establish this backup mirror to more than one primary disk set. Obviously you can’t back up more than one primary disk set at one time using this method, but suppose you have primary disk sets A and B. Once you finish establishing, splitting, and backing up primary disk set A, you can establish the backup mirror to primary disk set B, split it, then back it up again. Many people rotate their backup mirrors this way.
However, some storage vendors have a solution that’s less expensive than having a separate backup mirror for each primary disk set and is more elegant than rotating a single backup mirror between multiple primary disk sets: snapshots.
Again, let’s start with an enterprise storage array or a group of JBOD being managed by an enterprise volume manager. Recall that in the previous section “LAN-Free Backups,” some volume managers and storage arrays can create a virtual copy, or a snapshot. To another host that can see it, it looks the same as a backup mirror. It’s a static “picture” of what the volumes looked like at a particular time.
Creating the snapshot is equivalent to the step where you split off the backup mirror. There is no step where you must establish the snapshot. You simply place the application into a backup status, create the snapshot, and then take the application out of backup status. After that, the storage array or enterprise volume manager software makes the device visible to another host. You then perform the same steps as outlined in the previous section “Backing Up the Backup Mirror,” for importing and mounting the volumes on the backup server. Once that is accomplished, you can back it up just like any other device.
Similar to the backup mirror backup, the snapshot can typically be left around in case of recovery. However, most snapshot recoveries aren’t going to be as fast as the backup mirror recovery discussed earlier. Although the data can still be copied from disk to disk, the recovery may not be as instantaneous. However, since this section of the industry is changing so fast, this may no longer be the case by the time you read this.
It should also be noted that, unlike the backup mirror, the snapshot still depends on the primary disk set’s disks. Since it’s merely a virtual copy of the disks, if the original disks are physically damaged, the snapshot becomes worthless. Therefore, snapshots provide quick recovery only in the case of logical corruption of the data, since the snapshot still contains a noncorrupted copy of the data.
This variation on client-free backup represents a valid option for many. Depending on your configuration, this method can be significantly less expensive than a backup mirror, while providing you exactly the level of recoverability you need.
So far in this chapter, we have discussed LAN-free and client-free backups. LAN-free backups help speed backup and recovery by removing this traffic from the LAN. They allow you to purchase one or more really large tape libraries and share those tape libraries among multiple servers. This allows you to take advantage of the economies of scale that large libraries provide, as the cost per MB usually decreases as the size of a library increases. Backing up large amounts of data is much easier on the clients when that data is sent via Fibre Channel, instead of being sent across the LAN.[12] Therefore, LAN-free backups reduce the degree to which backups affect any applications that are running on the client.
Client-free backups also offer multiple advantages. They remove almost all application impact from the backup client, since all significant I/O operations are performed on the backup server. If you use a backup mirror for each set of primary disk sets, they also offer a virtually instantaneous recovery method. However, this recovery speed does come with a significant cost, since purchasing a backup mirror for each set of primary disk sets requires purchasing more disk capacity then would otherwise be necessary. If your primary disk set is mirrored, you need to purchase a backup mirror, requiring the purchase of 50% more disk. If the primary disk set is a RAID 4 or RAID 5 volume, you will need to purchase almost 100% more disk.[13]
What about snapshots? You may remember that we have discussed snapshots as an alternate way to provide instantaneous recoveries. However, snapshots (that have not been backed up to tape) only protect against logical corruption, and the loss of too many drives on the primary disk set results in a worthless snapshot. Therefore, client-free backups that use a backup mirror are the only backup and recovery design that offers instantaneous recovery after the loss of multiple drives on the primary disk set.
The remaining disadvantage to client-free backups is that you still need a server to act as a data path for the data to be backed up. Let’s say that you’ve got several large backup clients running the same operating system, and all of their storage resides on an enterprise storage array that can perform client-free backups. Chances are that you are now talking about a significant amount of storage to back up each night. In fact, many client-free backup designs use full backups every night, because every file they back up changes every night. This is the case with almost any database that uses “cooked” files on the filesystem for its data files and is always true of a database that uses raw devices for the same purpose. (While there are a few block-level incremental packages available, they aren’t yet widely used.) This means that if you back up five 2-TB clients, you are probably backing up 10 TB every night. Even though you have offloaded the data transfer from the data servers to the backup server, you will need a reasonably large server to back up 10 TB every night, and that server can have no purpose other than that.
With LAN-free backups, you back up the data without using the LAN. With client-free backups, you back up the data without using the client. What if you could transfer the data directly from disk to tape, without going through any server at all? If you could, your backups would be completely server-free. As with client-free backups, applications are almost completely unaffected, because few I/O operations are performed on the backup client.
One significant difference between client-free backups and server-free backups is that there are no homegrown server-free backup designs. The reasons for this will become clear as you see how deep into the filesystem and device driver levels you must go to accomplish the task.
A truly server-free backup has a data path that doesn’t include a server. It uses a server, of course, to control the process, but the data moves directly from disk to tape without going through any server’s CPU–including the backup server. There are three essential requirements of server-free backup. You must be able to:
In order to use a backup mirror setup to present a static copy to the backup application, you must perform the following steps:
If you wish to use a snapshot to present a static copy to the backup application, you need to perform only two steps:
WARNING:
Not all database applications permit online third-party backups.
The procedures performed during this step are essentially the same as those used for client-free backup. Therefore, if you need more detail about these steps, please refer to that section of this chapter.
Now that the backup application has a static view of the disks to back up, you must perform an additional step that is necessary because of the way you back up the data. LAN-free and client-free backups back up filesystem data in the traditional way–via the filesystem. However, the method server-free backup uses to transfer the data to tape has no knowledge of the filesystem. As you’ll see in the next step, it will simply transfer blocks of data to tape.
Therefore, prior to transferring these blocks of data to tape, you must create a map of which blocks belong to which files. That way, when the backup application is asked to restore a certain file, it knows which blocks of data to restore. Figure 4-15 illustrates how this works. File A consists of data blocks A, B, and C. File B consists of data blocks D, E, and F. Once the snapshot or split mirror has been created, these mappings remain constant until the split mirror or snapshot is created again. While it’s static, the backup application records the file associations for File A and File B. When the restore application asks for File A to be restored, the server-free backup software knows it needs to restore disk blocks A, B, and C.
The reason for the step in the previous section is that the main difference between client-free and server-free backups is the path the data takes on its way to tape–as illustrated in Figure 4-16. You’ll note that Figure 4-16 looks much like Figure 4-9 with a few minor changes. First, you no longer need the dedicated backup server that is seen to the right of the storage array in Figure 4-9. This means that Backup Server A takes the responsibility of quiescing the application that is writing to the disk. Second, the tape library and backup mirror (or snapshot) must be connected to a SAN with support for extended copy (xcopy). Finally, instead of sending the data to tape via the backup server (as shown in (4a) and (4b) in Figure 4-9), the backup software uses the SCSI extended copy command to send the data directly from disk to tape. This means, of course, that the data is being backed up on the raw-device level. Device-level backups are often referred to as image-level backups.
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
14.
If your backup software supports library sharing.
By W. Curtis Preston
When restoring data from a server-free backup system, you typically restore the data on the image level (file recoveries are rare but technically possible). Image level restores copy the entire image back from tape to disk. File level restores act just like nonserver-free restores, copying the file(s) you select from tape to disk. However, it’s important to mention that performing server-free backups doesn’t necessarily mean you are going to perform server-free restores. In fact, some server-free backup applications don’t support server-free restores at all. The ones that do support server-free restores do so only for image-level restores. To see why this is the case, let’s see how a server-free image-level restore works.
The server-free backup system in Figure 4-17 is the same one illustrated in Figure 4-16. Recall that we created a split mirror or snapshot, then used extended copy to back up that snapshot or mirror to tape. To reverse the process, you first have to create the snapshot to which you’re going to restore. Since the entire snapshot is going to be overwritten, it really doesn’t matter if it’s an older snapshot or one that was just taken. Regardless, the snapshot or split mirror must be created, as shown in (1) in Figure 4-17. Next, the server-free backup application can use the xcopy command to restore the backed up data directly from tape to disk, as shown in (3).
The next step is important. Just as you need to stop the application that was writing to the disk during backups, you need to do so during the restore. Otherwise, you’d really confuse things and corrupt the data. Also, depending on your operating system and filesystem type, you usually need to unmount the filesystem you’re about to restore. (This is shown as (4) in Figure 4-17.) The reason for this is that the next step is going to completely overwrite the blocks underneath the filesystem. If you do this while the filesystem is mounted, the filesystem is immediately corrupted. Finally, you can restore the image from the backup mirror or snapshot to the primary disk set. Once this has been accomplished, mount the filesystem and resume the application. (If the application is a database, you will probably need to perform a transaction log restore first.)
TIP:
A fully functional server-free backup system automates all the previous steps when you select an image-level restore.
As complicated as image-level restores may seem, file-level restores from server-free backups are even worse. In the top-left corner of Figure 4-18, you can see the original file-to-disk mapping shown in Figure 4-15. Below that, you can see that a snapshot or mirror is created of it, and that mirror is backed up to tape.
Time passes; files are added, deleted, and modified. Due to the way disk technology works, you can see that File A and File B still use the same blocks, but the blocks are now in completely different places on the disk. Now assume that File B has been deleted, as represented by the white disk blocks.
To perform an image-level restore, you first need to create a snapshot or mirror to restore to. This, of course, duplicates the new block layout of the disk. If you then restore the blocks of data associated with File B, you can see you’re restoring them to the wrong place. This overwrites the blocks of some other file, which corrupts that file.
Therefore, file-level restores from server-free backups need to be performed from the filesystem level, which is why file-level restores from server-free backups aren’t server-free at all. When you request a file to be restored, the backup software knows that the file was backed up via an image-level server-free backup. It then consults its file-to-block map made during that backup and knows which blocks of data it needs to read from tape. Those blocks are then passed to the client software running on the client where the file needs to be. It then reassembles the blocks into a file it can pass to the filesystem. The filesystem then places the file’s blocks in the appropriate site.
The advantages of server-free backups and restores are rather significant:
Cost is the main disadvantage in server-free backups. Server-free backups usually require some change in your primary disk set; it definitely needs to be on a SAN. Depending on the backup vendor you choose, you then need to do one of three things. You can:
Option 1 requires an enterprise storage array. Options 1 and 2 require additional disks, if you aren’t already using split mirrors for backup. Option 3 requires additional software. However, some backup products include the software to do this when you purchase the server-free backup add-on to their product. This software, of course, also comes at a price. The data mover can also exist as intelligence on the disk array, a Fibre Channel router, or another dedicated appliance on the SAN.
The other disadvantage to server-free backups is that the process is rather complex. It requires much more design on the part of the backup software company, since it needs to do the file-to-block mapping. When something goes wrong, there are more pieces of software to troubleshoot.
Which backup option is appropriate for your environment? The answer may be apparent if you ask yourself a few questions:
The easiest way to solve this problem is to turn the affected backup clients into device servers and attach them to one or more tape libraries. The difficulty comes when deciding which tape libraries to connect them to. Suppose that you’ve only got one client with this problem. You don’t necessarily want to buy a separate tape library just for one client. You can physically connect some of your first library’s tape drives to the affected client and use library sharing[14] to share the library between this backup client (which is now a device server) and the main backup server. Unfortunately, you probably sized that tape library based on how many drives you needed for the main server, and you are now taking away some of its resources. When you are in this situation, you will almost always benefit from LAN-free backups by connecting the main server and device server(s) to a SAN and using drive sharing to dynamically allocate drives to the servers that need them.
This scenario assumes you already encountered this situation, and have already physically allocated certain tape drives to certain servers. Depending on the number of servers and tape drives involved, you’d probably benefit from a LAN-free backup configuration.
If you’ve answered yes to the questions so far, you are a candidate for either client-free or server-free backups. The decision between the two is a bit more complex. Basically, you should get quotes for both systems, evaluate the cost of each in your environment, and then compare that cost to the benefits in Table 4-3.
|
|
LAN-free |
Client-free |
Server-free |
||
|---|---|---|---|---|---|
|
Backup client CPU, memory, and I/O load of TCP/IP processing to transfer data across LAN |
Significantly reduced (must still send metadata to backup server) |
Significantly reduced (must still send metadata to backup server) |
Significantly reduced (must still send metadata to backup server) |
||
|
Backup server CPU, memory, and I/O load of TCP/IP processing to transfer data across LAN |
Significantly reduced (must still receive metadata from backup client) |
Significantly reduced (must still receive metadata from backup client) |
Significantly reduced (must still receive metadata from backup client) |
||
|
Backup client CPU, memory, and I/O load of moving the data from disk to tape |
Same as LAN-based, since client is now responsible for moving split mirror or snapshot’s data from disk to tape |
No I/O load |
No I/O load |
||
|
Backup server CPU, memory, and I/O load of moving the data from disk to tape |
Significantly reduced (must still handle I/O of index operations) |
Same as LAN-based, because server is now responsible for moving split mirror or snapshot’s data from disk to tape |
Significantly reduced (must still handle I/O of index operations) |
||
|
Must client be on SAN? |
Yes (for drive sharing) |
Yes |
Yes |
||
|
Must client’s primary disk be on a SAN? |
No |
Yes |
Yes and SAN must support extended copy |
||
|
File-level recoveries |
Yes |
Yes |
|
Probably not |
|
|
|
Snapshot |
Backup mirror |
Snapshot |
Backup mirror |
|
|
Additional disk |
None |
Cache disk only |
Equivalent to usable disk on |
Cache disk only |
Equivalent to usable disk on |
|
Additional software |
Dynamic drive allocation |
Snapshot control, backup control |
Backup mirror |
Snapshot control, backup control with xcopy support |
Backup mirror |
|
Additional server |
No |
Yes |
Yes |
No |
No |
|
Allows homegrown solutions |
No |
Yes |
Yes |
No |
No |
Author
W. Curtis Preston has specialized in designing backup and recovery systems for over eight years, and has designed such systems for many environments, both large and small. The first environment that Curtis was responsible for went from 7 small servers to 250 large servers in just over two years, running Oracle, Informix, and Sybase databases and five versions of Unix. He started managing this environment with homegrown utilities and eventually installed the first of many commercial backup utilities. His passion for backup and recovery began with managing the data growth of this 24×7, mission-critical environment. Having designed backup systems for environments with small budgets, Curtis has developed a number of freely available tools, including ones that perform live backups of Oracle, Informix, and Sybase. He has ported these tools to a number of environments, including Linux, and they are running at companies around the world. Curtis is now the owner of Storage Designs, a consulting company dedicated entirely to selecting, designing, implementing, and auditing storage systems. He is also the webmaster of www.backupcentral.com.
Footnotes:
1.
This term may be changed in the near future, since iSCSI-based SANs will, of course, use the LAN. But if you create a separate LAN for iSCSI, as many experts are recommending, the backups will not use your production LAN. Therefore, the principle remains the same, and only the implementation changes.
2.
As mentioned later in this chapter, SCSI devices can be connected to more than one host, but it can be troublesome.
3.
This is actually a high rate of change, but it helps prove the point. Even with a rate of change this high, the drives still go unused the majority of the time.
4.
1.575 TB ÷ 8 hours ÷ 60 minutes ÷ 60 seconds = 54.6 MB/s
5.
There are several tape drives capable of these backup speeds, including AIT-3, LTO, Mammoth, Super DLT, 3590, 9840, and DTF.
6.
20 minutes × 24 hosts = 480 minutes, or 8 hours
7.
These are Unix prices. Obviously, Windows-based cards cost much less.
8.
Although it’s possible that some software products have also implemented a third-party queuing system for the robotic arm as well, I am not aware of any that do this. As long as you have a third-party application controlling access to the tape library and placing tapes into drives that need them, there is no need to share the robot in a SCSI sense.
9.
Network Appliance filers appear to act this way, but the WAFL filesystem is quite a bit different. They store a “before” image of every block that is changed every time they sync the data from NVRAM to disk. Each time they perform a sync operation, they leave a pointer to the previous state of the filesystem. A Network Appliance snapshot, then, is simply a reference to that pointer. Please consult your Network Appliance documentation for details.
10.
It was the Microsoft’s partnership with Veritas that finally made this a reality. The volume manager for Windows 2000 is a “lite” version of Veritas Volume Manager.
11.
Prior to 9i, this was done with the suvrmgr command, but this command has been removed from 9i.
12.
There are vendors that are shipping gigabit network cards that offload the TCP/IP processing from the server. They make LAN-based backups easier, but LAN-free backups are still better because of the design of most backup software packages.
13.
It’s not quite 100%, since the second stripe doesn’t have to be a RAID 5 set. If it were simply a RAID 0 set, you’d need about 90% more disk than you already have.
Enterprise Storage Forum offers practical information on data storage and protection from several different perspectives: hardware, software, on-premises services and cloud services. It also includes storage security and deep looks into various storage technologies, including object storage and modern parallel file systems. ESF is an ideal website for enterprise storage admins, CTOs and storage architects to reference in order to stay informed about the latest products, services and trends in the storage industry.
Property of TechnologyAdvice. © 2025 TechnologyAdvice. All Rights Reserved
Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.
