Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Filling the File System
One of the keys to the testing is how the file system is filled. This can be a very time consuming process because you must create all of the files in some sort of order or fashion. For this testing, fs_mark was used. Ric Wheeler at Red Hat has been using it for testing file systems at very large scales (over 1 billion files). Fs_mark wasn't used for testing the file system in this article, but rather, it is used to fill the file system in a specific fashion. It uses one or more base directories and then creates a specified number of subdirectories underneath them that are filled with files. You might think of this as a single-level of subdirectories. It is much more complicated to create specific subdirectory depths and number of files since that configuration depends on the specific users and situation. You could also use some sort of random approach with the hope that a random distribution approximates a real-world situation. It is virtually impossible to have a representative file system tree that fits most general situations, and the single-level deep directory tree used here should represent one extreme of file systems -- a single subdirectory level.
One of the nice features of fs_mark is that it is threaded so that each thread produces its own unique directory structure with a single layer of subdirectories underneath a base directory that contains a fixed number of files. Fs_mark also allows you to specify the number of files per thread so that you can control the total number of files. Although the server has eight total cores, running eight threads (one per core) it resulted in the OS swapping. When the number of threads is reduced to three, the server did not swap, and the file creation rate was much faster than running eight threads with swapping.
The goal for all fs_mark commands was to fill the file system to the specified number of files while filling about 50 percent of the file system. The following fs_mark command lines were used to fill the file system for 72TB:
- ./fs_mark -s 400000 -L 1 -S 0 -n 35000000 -D 35000 -N 1000 -t 3 -k -d /mnt/test
- ./fs_mark -s 800000 -L 1 -S 0 -n 17000000 -D 17000 -N 1000 -t 3 -k -d /mnt/test
- ./fs_mark -s 4000000 -L 1 -S 0 -n 3400000 -D 3400 -N 1000 -t 3 -k -d /mnt/test
The commands for filling the 38TB file systems were:
- ./fs_mark -s 200000 -L 1 -S 0 -n 35000000 -D 35000 -N 1000 -t 3 -k -d /mnt/test
- ./fs_mark -s 400000 -L 1 -S 0 -n 17000000 -D 17000 -N 1000 -t 3 -k -d /mnt/test
- ./fs_mark -s 2000000 -L 1 -S 0 -n 3400000 -D 3400 -N 1000 -t 3 -k -d /mnt/test
Notice that the number of files per directory is a constant (-N 1000 or 1,000 files).
After the file system was filled using fs_mark, it was unmounted, and the file system check was run on the device. In the case of xfs, the command is,
/sbin/xfs_repair -v /dev/md1
For ext4, the file system check was,
/sbin/e2fsck -pfFt /dev/md1
Notice that the device /dev/md1 was the target in both cases.