Enterprise Storage Forum content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Deduplication was the province of a couple of startups such as Data Domain about ten years ago. Since then, it has almost become ubiquitous. In fact, it is often included in other products as a feature. That said, there are still plenty of good deduplication tools out there.
Quantum’s DXi line of disk-based systems with built-in deduplication consists of several products. The DXi6900, for example, is built for enterprise-wide backup, disaster recovery and data protection. It provides up 510 TB of usable capacity in a 2U rack space. It provides 10GbE, 1GbE and 8 GB Fibre Channel (FC) connectivity.
EMC Data Domain utilizes high-speed, variable-length deduplication, which segments data to achieve dedupe efficiencies of 10x to 30x on average. Additionally, Data Domain Boost distributes parts of the deduplication process to the client or application server, so that the workload is spread across the server and Data Domain. Only unique data is sent across the network to the Data Domain system, enabling faster backup performance and more efficient resource utilization. The DD Boost ecosystem boasts over 15 applications – including the majority of the backup software market as well as enterprise applications like Oracle, Microsoft SQL and SAP. It also enables DBAs/app owners to control backup from their own utilities such as Oracle RMAN.
“DD Boost has changed the game for deduplication and ended the debate of client vs. target deduplication by providing the best of both worlds of deduplication,” said Peter Smails, Vice President of Marketing for Core Technologies at EMC. “It’s become the primary way applications write to Data Domain – including our own Avamar and NetWorker.”
He said that the majority of customers who purchase Avamar use its software capabilities (VMware integration, NAS backup) and write that data to Data Domain (using DD Boost). Therefore, Avamar deduplication is less relevant today as most Avamar users now take advantage of Data Domain.
Assigra Cloud Backup includes a wealth of features. As well as deduplication, it offers archiving, Continuous Data Protection (CDP), compression, encryption, VM replication and backup. On the dedupe side, it provides local as well as global deduplication i.e. it deduplicates across multiple sites at the same time.
NetBackup used to belong to Symantec but that company spun off its various backup products into Veritas. As such, NetBackup appliances and software include a deduplication engine for physical as well as virtual environments.
ASG acquired its deduplication and data protection capabilities from its purchase of Atempo four years ago. It provides source-based block level deduplication and backs up servers, desktops and laptops in remote offices and branch offices.
But the trend appears to be away from dedicated deduplication products with this function instead being part of a broader product suite. Probably the best example of this is Commvault Simpana data protection software. This platform includes deduplication, automated provisioning, management of cloud infrastructure (support for Amazon Web Services and Microsoft Azure), recovery, software snapshots, secure file sharing, search, endpoint data protection and virtual machine resource management.
“Today’s data management complexities, particularly the adoption of cloud technologies, are prompting more enterprises to look for ways to quickly and easily modernize their environments,” said Dave Simpson, Senior Analyst, Storage for the 451 Group. “CommVault’s, modular approach allows customers to address specific pain points, with the flexibility to upgrade to the full Simpana software suite.”
Similar to CommVault, HP folds its deduplication capabilities into other products. The problem with the current HP Enterprise site (HPE), however, is that you can’t find any of its deduplication tools or features easily. They appear to be lost in a sea of other offerings. Some detective work uncovered the HP Data Protector which is a backup and recovery product that includes deduplication, analytics and optimization features. It provides a platform for standardizing the backup and recovery of information spread across locations, applications, formats, storage platforms, operating systems and hypervisors. HP StoreOnce Backup also includes deduplication.
Bacula is a backup, restore and disaster recovery suite that includes Bacula Enterprise Deduplication Volumes. Deduplication Volumes store data on disks by aligning file boundaries to the block boundary of the underlying file system. Metadata, which does not align, is separated into a special metadata volume. Within the data volume, the space between the end of one file and the start of the next block boundary is left empty. Since every file begins on a block boundary, redundant data within files will deduplicate well using ZFS’s fixed block deduplication.
