Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
The last couple of years have seen a surge in file system development.
Sun (NASDAQ: JAVA) open sourced ZFS and acquired Lustre. Quantum (NYSE: QTM), Panasas, IBM (NYSE: HPQ), HP (NYSE: HPQ), NetApp (NASDAQ: NTAP) and Red Hat (NYSE: RHT) have all boosted their file system development efforts. And NFS is getting a major overhaul in the form of pNFS. Not surprisingly, all this development effort has stimulated work in open source communities to come up with higher performance file systems for Linux.
Enter a couple of developments on the Linux front: the BTR File System known as Butter FS or B-tree FS and ext4.
"Ext4 is an update to the widely-used ext3 that upgrades capacities in a number of areas, tweaks performance, and so forth," said Gordon Haff, an analyst at Illuminata. "In the relatively near-term, ext4 is the next new file system that we're likely to see widely deployed on Linux."
Although it was developed in 2006, ext4 has only really been stabilized over the last year and is only now beginning to see any volume of deployment. Versions of ext4, for instance, appear in recent releases such as Fedora. This Linux file system adds larger storage capacities (volumes up to 1 Exabyte) as well as various performance tweaks.
Ext4 is very much an evolution of the popular ext3 file system. Ext3 is limited to 16TB of storage, while ext4 has been designed to handle even larger file systems. One data integrity feature that has been added to ext4 is support for checksums on its internal journal transactions, which gives it some additional robustness in the face of storage errors.
But ext4 isn't so significant a development as BTRFS, at least in terms of potential. BTRFS may eventually pose more of a threat to ZFS and others, though it has yet to be finalized. It brings online defragmentation support, a mode just for solid state drives (SSDs), logging for copy-on-write, and various compression, mirroring, stripping and snapshot bells and whistles.
"BTRFS is a next-generation file system, which broadly means that it uses a different administration model from traditional file systems," said Haff.
What he means it that it moves beyond the administration and availability models used by current file systems. He gives an example: it essentially wraps volume management historically a separate product into the file system itself. It also builds in reliability and availability features such as snapshots.
BTRFS, then, is much more aggressive than ext4 with regard to data storage. It has been designed to subsume some of the features that are normally done by logical volume managers (LVM) and RAID hardware, does checksums for both its internal metadata and user data, and has built-in support for snapshots (like an LVM). Several of these features can be done with ext4, but require interacting with both the file system and the logical volume manager.
Interestingly, BTRFS was initially developed by Oracle (NASDAQ: ORCL) which is acquiring Sun but is now licensed under the GPL and is thoroughly in the open source community, open for contribution from anyone. It is described on its own project pages as "a new copy on write file system for Linux aimed at implementing advanced features while focusing on fault tolerance, repair and easy administration."