dcsimg

Open Source Storage: 64 Applications for Data Storage

NewsletterIconSTORAGE DAILY

Want the latest storage insights?

Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure

SHARE
Share it on Twitter  
Share it on Facebook  
Share it on Google+
Share it on Linked in  
Email  

As data storage needs continue to grow and many organizations move toward software-defined infrastructure, more enterprises are using open source software to meet some of their storage needs. Projects like Hadoop, Ceph, Gluster and others have become very common at large enterprises.

Home users and small businesses can also benefit from open source storage software. These applications can make it possible to set up your own NAS or SAN device using industry-standard hardware without paying the high prices vendors charge for dedicated storage appliances. Open source software also offers users the option to set up a cloud storage solution where they have control over security and privacy, and it can also offer affordable options for backup and recovery.

The list below features 64 open source storage projects that are among the best options available for enterprises, SMBs and individual users. Please note that this is not a ranking. Entries are organized into categories and then alphabetized within the categories.

If you know of additional open source storage applications that you believe should be on our list, feel free to note them in the Comments section below.

NAS/SAN Software

1. Ceph

Red Hat's Ceph offers unified object and block storage capabilities. It's a distributed storage solution that boasts excellent performance, scalability and reliability. Well-known users include Cisco, DreamHost, CERN, Bloomberg, and Deutsche Telekom. Operating System: Linux

2. CryptoNAS

This project aims to simplify the process of setting up an encrypted fileserver. It comes in a live-CD package or a server package that adds a web front-end. Operating System: Linux

3. ESOS

Short for Enterprise Storage OS, ESOS is a Linux distribution for setting up a storage array on your own hardware. Commercial support is available. Operating System: Linux

4. FreeNAS

FreeNAS claims to be the "the World's #1 storage OS with over 10+ million downloads." It counts the United Nations, the University of Florida, the Salvation Army, Reuters, Michigan State University, the Department of Homeland Security and many other organizations among its users. It can be installed on nearly any hardware to turn it into a network attached storage (NAS) device. Paid, supported enterprise solutions based on the same technology are available under the TrueNAS brand name. Operating System: FreeBSD

5. NAS4Free

Another option for do-it-yourself NAS, NAS4Free claims to be "the simplest and fastest way to create a centralized and easily-accessible server for all kinds of data." Key features include the ZFS file system, software RAID (levels 0, 1 or 5) and disk encryption. Operating System: FreeBSD

6. Openfiler

As a unified storage solution, Openfiler aims to combine the best features of NAS storage with the best features on SAN storage. Key features include high availability/failover, block replication and Web-based management. Its users include Motorola, Pratt & Whitney, Bill Me Later and the London Metropolitan Police. A paid commercial edition is available. Operating System: Linux

7. OpenMediaVault

Based on Debian Linux, OpenMediaVault describes itself as a "next-generation network attached storage (NAS) solution." It was designed to provide NAS for home users or small offices. It offers a Web-based administration console and includes software RAID capabilities. Operating System: Linux

8. Turnkey Linux File Server

The Turnkey Linux project offers images for setting up Linux-based servers for a variety of purposes, including an image for creating a simple NAS device. It includes support for SMB, SFTP, NFS, WebDAV and rsync file transfer protocols. Operating System: Linux

Filesystems

9. Btrfs

A joint project supported by Facebook, Fujitsu, Intel, the Linux Foundation, Netgear, Novell, Oracle, Red Hat and others, Btrfs is a copy on write (CoW) filesystem for Linux. It focuses on "fault tolerance, repair and easy administration." Operating System: Linux

10. Ext4

Included in most popular Linux distributions, Ext4 supports file systems up to 1 EB in size with up to 16 TB per file. Other key features include extents, multiblock allocation, delayed allocation, Fast fsck, journal checksumming, "no journaling" mode, online defragmentation and more. Operating System: Linux

11. GlusterFS

A Red Hat project, GlusterFS is a highly scalable file system built for applications like media streaming and big data analytics. Professional support is available through third-party vendors. It has a large and active user community, and the website includes links to many Gluster-related blogs. Operating System: Linux

12. Lustre

Built to handle the needs of high-performance computing (HPC) environments, Lustre is a scalable parallel file system. It was first developed at Carnegie Mellon University, and its first users included various U.S. Department of Energy National Laboratories. The latest version, released in April, supports Data on MDT, file-level redundancy, lock ahead and more. Operating System: Linux

13. ZFS

Also incorporated into many Linux distributions, including Arch, Debian, Fedora, Ubuntu and others, ZFS is another highly scalable file system. It includes compression, protection against data corruption, snapshots, RAID support and more. Operating System: Linux, macOS, FreeBSD

RAID

14. DRBD

DRBD is an open source solution for building high-availability storage clusters based on RAID-1. Commercial products, including software-defined storage, disaster recovery and high availability solutions based on the same technology, are available through project sponsor Linbit. Operating System: Linux

15. Mdadm

Built into the Linux kernel, mdadm makes it easy to create, manage and monitor storage arrays. It can also move spares between RAID arrays. More information is also available on the Linux RAID wiki. Operating System: Linux

16. Raider

Raider is a bash shell script that allows users convert any Linux disk into a RAID array with software RAID 1, 4, 5, 6 or 10. It works with many of the most popular Linux distributions, including Ubuntu, Debian, OpenSUSE, Fedora, Linux Mint and others. Operating System: Linux

17. RaidEye

RaidEye isn't so much a tool for creating RAID devices as a utility for monitoring RAID volumes. It works with the RAID capabilities built into macOS and notifies users of problems via a sound alarm, popup dialog and email. Operating System: macOS

18. SnapRAID

SnapRAID arrays can recover from up to six disk failures. The tool was built for home media servers or other environments with a lot of big files that rarely change. Key features include data hashing, the ability to recover deleted files and no lock-in. Operating System: Windows, Linux, macOS

Backup and Synchronization

19. AMANDA

The Advanced Maryland Automatic Network Disk Archiver, or AMANDA, is a popular network backup solution that can save data from Linux, Unix or Windows systems to hard drives, tape or optical media. It was last updated in December 2017. Zmanda, which sponsors the project, offers commercial products based on the same technology. Operating System: Windows, Linux, macOS.

20. Areca Backup

Designed for personal use, Area is a simple but versatile backup solution. Key features include delta backup, compression, encryption, filters, as-of-date recovery and more. Operating System: Windows, Linux

21. BackupPC

Robust enough for enterprise use, BackupPC backs up data from Linux and Windows systems to disk. Noteworthy features include a unique pooling scheme, optional compression, a web interface and support for mobile devices. It claims to be highly configurable and easy to install and maintain. Operating System: Windows, Linux

22. Bacula

Another option for enterprises, Bacula is a network backup solution that aims to be easy to use and very efficient. It claims to be the most popular open source backup program. Commercial support and services for the solution are available through Bacula Systems. Operating System: Windows, Linux, macOS

23. Bareos

Forked from Bacula, Bareos is a popular open source backup option that is under very active development with the latest version released in February 2018. The Bareos.com website offers paid support and services for the tool. Operating System: Windows, Linux, macOS

24. Box Backup

This "completely automatic" backup solution creates backups continuously and can also create snapshots when desired. It includes encryption and optional RAID capabilities, and old file versions and deleted files remain available. Operating System: Windows, Linux

25. BURP

Short for "BackUp And Restore Program," BURP is a network backup solution. It offers a choice of two different protocols: one based on librsync (see below) and one that uses variable length chunking for inline deduplication. It is designed to be easier to configure than some other open source solutions, and it can do delta backups. Operating System: Windows, Linux

26. Clonezilla

Designed to replace Acronis True Image or Norton Ghost, Clonezilla is useful for both system deployment and backup and recovery. It comes in two flavors: live for standalone systems and SE for network backup or cloning multiple systems at once. The SE version can clone 40 or more systems at once. Operating System: Linux

27. Create Synchronicity

Powerful but lightweight, this backup tool takes up only 220KB of space on your hard drive. It supports multiple languages, has an intuitive interface and includes a scheduler. It is also helpful for syncing photos, music and other files across devices. Operating System: Windows

28. DAR

Disk Archive, a.k.a. DAR, is an older command-line tool for backup, but it is still being updated regularly with the most recent version released in April 2018. For those who prefer a GUI, one is available through the DarGUI project. Operating System: Windows, Linux, macOS

29. DirSync Pro

This "small but powerful," utility offers incremental backup, filtering and scheduling capabilities. It also boasts an intuitive interface, and it offers the ability to analyze two sets of files or folders and detect the changes between them. It also includes a helpful real-time synchronization option. Operating System: Windows

30. Duplicati

Duplicati works with cloud storage services like AWS S3, Microsoft OneDrive, Google Drive and Box to create backups with AES-256 encryption. It does a full backup on first use and incremental backups after that; it also offers data deduplication capabilities. Operating System: Windows, Linux, macOS

31. FOG

FOG offers cross-platform cloning and imaging capabilities plus remote management for networks of any size. It offers support through forums and a wiki. Operating System: Linux, Windows, macOS.

32. FreeFileSync

A tool for standalone systems, FreeFileSync aims to save users time when setting up and running backups. It is cross-platform and includes 64-bit support. Tutorials and a manual are available on the website. Operating System: Linux, Windows, macOS

33. FullSync

Although it was designed to help web developers push updates to their sites, FullSync can also be used by anyone to create backups. Key features include multiple modes, flexible rules, buffered filesystems, support for multiple file transfer protocols and more. Note that development on this project has slowed somewhat as it hasn’t been updated since April 2016. Operating System: Linux, Windows, macOS

34. Grsync

Grsync takes the older rsync synchronization tool and adds an easy-to-use GUI. Noteworthy features include unlimited sessions, highlighted errors, batch capabilities, simulations, support for multiple languages and more. Operating System: Linux, Windows, macOS

35. Mondo Rescue

For Linux and FreeBSD only, Mondo Rescue is a disaster recovery solution that supports tape, disk, network or optical media backups. According to its website, its users include "Lockheed-Martin, Nortel Networks, Siemens, HP, IBM, and dozens of smaller companies." The most recent update was released in April 2016. Operating System: Linux, Free BSD

36. Partimage

This tool saves partitions of drives as image files, making it useful for backup or installing the same image on multiple systems. It can run across networks or on a standalone PC. It can also be used to create a SystemRescueCD. Operating System: Linux

37. Redo

Redo boasts that its bare-metal restore capabilities can get a crashed system back up and running in as little as 10 minutes. It's very easy to use and can also recover deleted images and files. Operating System: Windows, Linux

38. Rsync

Rsync is a Unix-based file-transfer utility with synchronization capabilities that make it suitable for creating backups or mirroring. It's a useful tool but is best used by advanced users. The most recent version was released in January 2018. Operating System: Linux, Windows, macOS

39. Synkron

While this app is focused primarily on synchronization, it can be used for creating backups as well. Key features include analysis capabilities, blacklisting, restores and cross-platform support. Documentation is available in both German and English. Operating System: Windows, Linux, macOS

40. Unison

Like Synkron, Unison is a file synchronization tool. It can copy files between any two systems connected to the internet, and it has features in common with source code management tools as well as with backup utilities. Its advantage over some other synchronization tools is that it can combine two sets of files where both sets have undergone changes. Operating System: Windows, Unix

41. UrBackup

This client-server backup solution does both image and file backups. It promises "both data safety and a fast restoration time." It makes backups while the system is in use without interrupting normal operation. Operating System: Windows, Linux

42. Weex

The Weex developers intended it primarily as a tool for pushing content to websites, but it can also be used to synchronize or backup files. It supports FTP file transfer and uses caching to speed data transfer. Operating System: Windows, Linux

Online/Cloud Data Storage

43. CloudStack

This Apache Foundation project is open source cloud computing platform that includes cloud storage capabilities. Noteworthy features include compute orchestration, network as a service capabilities, user and account management, resource accounting and support for multiple hypervisors. Operating System: Windows, Linux

44. CloudStore

This Dropbox alternative synchronizes data between a system and online storage. It promises strong encryption, password-less authentication, flexible synchronization, fast setup and auto-resumes for interrupted data transfers. Operating System: Linux

45. Cozy

Cozy is both an open source project for storing personal data online and a free service for managing and securing sensitive data. Note that the free hosting is for up to 5GB of data; additional storage will require a fee. Operating System: Linux

46. FTPbox

Want to set up your own cloud storage server? FTPbox makes it easy to be your own cloud provider, with all files transferred via FTP. Operating System: Windows

47. OpenStack

Probably the best-known open source cloud computing platform, OpenStack offers a complete operating system for controlling compute, networking and, of course, storage in the cloud. It incorporates three sub-projects related to storage: Cinder, Swift and Manila. Operating System: Windows

48. Perkeep

Formerly known as Camlistore, Perkeep describes itself as "a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era." It's still under very active development and will require some technical knowhow to use. Operating System: Linux

49. Pydio

Downloaded more than a million times, Pydio counts the University of Cambridge, Seagate, Guitar Center, Washington State University and Nikon among its users. It offers cloud-based file management and sharing. A paid enterprise distribution is available. Operating System: Windows, Linux (Android and iOS clients available)

50. Rockstor

Rockstor makes it possible to create your own NAS or cloud storage solution based on Linux and BTRFS. It comes in both personal and SMB versions. Operating System: Linux

51. SeaFile

SeaFile describes itself as "an enterprise file hosting platform with high reliability and performance." You can download the code for free or use the paid pro edition that includes support. Operating System: Windows, Linux, macOS, Android, iOS

52. SparkleShare

SparkleShare creates a special folder on your system that is automatically synchronized with a host folder stored on your server or in the cloud. It includes encryption capabilities, and it is a good option for collaborating on documents that change frequently. Operating System: Windows, Linux, macOS

53. StackSync

Create your own scalable personal cloud with StackSync. It encrypts all data on the client side and works with cloud storage services or your own server. Operating System: Windows, Linux

54. Syncthing

Like many of the other projects in this category, Syncthing offers an alternative to Dropbox. It keeps data completely private with encryption and authentication requirements. Operating System: Windows, Linux, macOS

Storage Management

55. Libvirt Storage Management

Libvirt is an API for creating storage pools and volumes on a host system. It supports a wide variety of storage pool types including directory, filesystem, network filesystem, logical volume, disk, iSCSI, SCSI, Gluster, ZFS and more. Operating System: Linux

56. openAttic

This tool offers management and monitoring capabilities for the Ceph distributed storage platform. It offers a dashboard, as well as tools for managing pools, block devices, iSCSI, NFS, Ceph Object Gateway and Ceph nodes. Operating System: Linux

Distributed Storage/Big Data Tools

57. Alluxio

Alluxio (Formerly known as Tachyon) describes itself as "open-source memory-speed virtual distributed storage." It works with tools like Spark, Hadoop, Flink, Zeppelin and Presto to speed performance on big data queries. Operating System: Linux, macOS

58. Hadoop

Nearly synonymous with big data, Hadoop is a widely used open source distributed storage platform for processing data. It is an Apache Foundation project, and the organization also oversees dozens of related projects. Operating System: Windows, Linux, macOS

59. HPCC

This Hadoop alternative also offers distributed storage and massive scalability. Paid enterprise services are available. Operating System: Linux

60. Sheepdog

The Sheepdog website describes this project as "a distributed object storage system for volume and container services and manages the disks and nodes intelligently." It supports snapshotting, cloning and thin provisioning, and it is compatible with OpenStack Swift and Amazon S3. Operating System: Linux

Compression

61. 7-zip

This archiving utility can compress files 30 to 70 percent smaller than WinZip can, and 2 to 10 percent smaller than most other zip programs. It includes encryption capabilities, self-extracting files, a powerful file manager and support for a wide variety of compression formats. Operating System: Windows, Linux, macOS

62. ArcThemALL

ArcThemALL can compress to UPX, ZIP or 7Z formats and extracts 33 other types of archive files. It can also create self-extracting archives and includes AES-256 encryption capabilities. Operating System: Windows

63. Keka

For macOS, Keka is a port of 7-zip. It also compresses to Zip, Tar, Gzip, Bzip2, DMG and ISO formats. Operating System: macOS

64. PeaZip

Amazingly versatile, PeaZip can open more than 180 different types of archived files, and it writes to seven, including ZIP and 7Z. It also includes security features like strong encryption, encrypted password manager, two-factor authentication, secure delete and file hashing. Operating System: Windows, Linux, macOS.

Submit a Comment

 

People are discussing this article with 0 comment(s)