Successfully Modeling and Simulating Systems, Part 2 Page 3
The second part of the modeling processes is ensuring that the model is accurate. As part of this process, the modeler needs to develop and/or work with someone to develop a methodology for testing the model. This process is commonly known as model calibration. The person developing the tests must have a detailed understanding of how the hardware and/or software work. Take a simple host bus adapter (HBA) as an example. If someone wanted to create a generalized model of a Fibre Channel HBA, they would have to take into account:
- 1 Gbit or 2 Gbit interface
- Latency through the HBA based on distance to the switch
- The number of commands that can be queued in the HBA
- How the commands are processed:
- Sorted or non-sorted
- Number of buffer credits needing writes
- Size of the command queue
- Number of targets that will be processed in the HBA (LUNs the HBA will be writing to)
- Failover issues
As you can see, for a fairly simple piece of hardware you have a great deal of work ahead of you just in modeling the HBA. Add to this the issues to consider for the software stack to the HBA, which include:
- Application and type of I/O being done
- C library if applicable
- System calls
- Operating System settings and tunables
- File System cache, layout settings, and tunables
- Volume manager layout settings and tunables
You can easily see that the complexity of what to test and how to test the various components of the model can be difficult and requires significant expertise.
Developing the tests that can measure each of the individual activities is often very difficult given the interactions that happen within the system between the various hardware and software components parts and services. In most cases, you cannot test a single device by itself; for example, when testing an HBA, even using just the raw device, you will also be testing a disk or RAID.
What I have found works best are reducing the variables and using different hardware. So, for example, when testing an HBA, it might be wise to set up a LUN about the same size as the RAID cache, and tune the cache to try to keep all of the data in cache. This way you reduce the effect of the disks and the RAID performance, and will likely have a more accurate representation of the HBA performance as a result. You can also change the RAID to another RAID type to see if the performance is different.
So in general, it is best to start with the characterization of the smallest and simplest pieces of hardware and software, and then move on to the more complex parts. The key to any characterization project is an understanding in reasonable detail of how the component being characterized works. In most cases, you really do not completely know how it is going to work, so you can expect to have a few surprises from the model, and you can expect to gain a great deal of insight into the inner workings of the project.