Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
Applications Programming Issues
Making requests that are not on 512-byte boundaries can cause serious performance problems for the system. There are a few very important "dos and don'ts" to ensure system performance using both system calls and/or the C Library package. Every major database uses system calls for I/O and each has internal caching mechanisms which bypass the buffer cache, so there's no need to worry about databases. The following table summarizes I/O types and the tradeoff between the these methods:
|I/O request and/or structure||System Calls||C Library Package|
|Sequential on 512-byte boundaries||If you can make the request size large (>256 KB) this is the best method||Only use if you cannot use system calls. You should use setvbuf(3) function to make user buffer size large so that requests are larger (>512 KB) than the record|
|Sequential not on 512-byte boundaries||Do not use unless you can modify code to pad data or change the requests to be on 512-byte boundaries||Shines over system calls if you can make buffer size at least 2x greater than request size rounded to the next 512-byte boundaries. Even for 1x the buffer size|
|Random I/O on 512-byte boundaries||System calls are the fastest, as a single request is done in one I/O||Never make buffer size larger than request size, as you will be making I/O requests in the buffer size reading data you will not use. If you need to use the type of I/O, ensure that the buffer is exactly the size of the request|
|Random I/O not on 512-byte boundaries||Do not use unless you can modify code to pad data or requests to be on 512-byte boundaries||Shines over system calls if the buffer size is exactly the size of the I/O request rounded to the next multiple of 512-bytes. Do not make the buffer size greater than the request size rounded up|
In general, changing the request size for a program that does sequential I/O using the C library package is far simpler than changing programs that make direct system calls. All the user must do is add a single line after the open to call the setvbuf(3) library call. For programs that make system calls often, this requires major rewrites to the program and data restructuring to make larger requests. Often this has implications on the program's computational structure.
This is typically a much more difficult problem. On rare occasions the solution is very simple. Sometimes I have seen that the file being opened is small by today's standards and can actually fit in memory. I had one program that I worked on a few years ago that used a 40 MB random I/O file and which opened and closed the C library package 100,000s of times, so it was never kept in memory. A simple change was made to use setvbuf(3) to keep it in memory and removed the open/close. The application performance improvement was over 300x and CPU time was significantly reduced.
The issues with random I/O and large files are a bit more complex. You never want to make requests larger than the physical request from the application. If the requests begin and end on 512-byte boundaries, then systems calls are your best choice. If they do not begin and end on 512-byte boundaries, then using the C library and setting the buffer to the request size rounded to the next 512-byte boundaries is a far better choice given the read-modify-write that will otherwise be required for the systems calls.
It is important to note that random I/O is not always as random as you think. Often applications perform what I call randomly sequential I/O. A number of requests are made sequentially and a seek request is made, and I/O is then done sequentially again. From what I have seen, this is very common in databases, search engines, and a number of scientific applications. We will discuss this further when we talk about RAID cache in a few months.
I/O at the operating system level is not overly difficult to understand given the current hardware restrictions of I/O beginning and ending of 512 byte boundaries. It is a very binary rule -- either the application does I/O on those boundaries or the system does it for you. Performance plays a major role in determining the most efficient use of systems calls and how they pass data to and from the file system.