Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
To help explain the trade-offs, let's begin by examining various storage solutions and how often they might encounter an error.
One of the most prevalent risks is hitting what is called the "hard error rate." This is the number of bits that can be read before a sector will not be able to be read. This hard error rate varies a great deal across the storage options.
|Device||Hard Error rate in bits||Equivalent in bytes||PB equivalent|
|LTO and Enterprise SAS SSDs||10E17||1.25E+16||11.10|
An alternative way to think about this data is to consider how long it would take me to encounter an unreadable sector if I read the data at the full rate of the device.
Henry's second table showed the amount of time it takes (in hours) to encounter a read error when you use more than one or more devices and start reading from those devices at maximum speed (he varied the number of devices from 1 to 200). Table 2 below reproduces his data and adds two more columns for 250 devices and 300 devices.
|Hours to reach hard error rate at sustained data read rate|
|Enterprise SAS/FC 3.5 inch||2,759.5||275.9||55.2||27.6||13.8||11.0||9.2|
|Enterprise SAS/FC 2.5 inch||1,965.2||196.5||39.3||19.7||9.8||19.7||9.8|
|Some Enterprise SAS SSDs||7,884.2||788.4||157.7||78.8||39.4||31.5||26.3|
These two tables illustrate that if you use SATA or SAS disks, the upper bound on capacity before you encounter an unreadable sector is around 1 PB and the lower bound is about 110 TB. And even with this data, it is very difficult to give any kind of estimate of when you could encounter the hard error because it becomes so configuration dependent.
Encountering a hard error usually means that the RAID controller flags the drive as bad and then starts a rebuild. To get access to the data you tried to read, you need to wait for the rebuild to finish. The amount of time to complete a rebuild can be fairly long, but again depends on the configuration.
During this time, all of the remaining drives have to be read, increasing the probability of encountering another hard read error and possibly losing the RAID group. Consequently, you really need to have more than one copy of the data in the archive.
The number of copies also remains an open question as Henry's article points out. But at a minimum, you need two copies and may even need three copies (I haven't heard of more than three copies yet unless it is a globally distributed archive). At a minimum this means you need two to three times the capacity of the archive in storage hardware alone. If you want a 1 PB archive, you will need 2-3 PB in storage capacity.