I have long been a proponent of FCoE (Fibre Channel over Ethernet). At one time, before the 2008 economic meltdown, I thought that paradigm-shifting technologies changes such as object storage and FCoE were on our doorstep.
It hasn’t worked out exactly as I envisioned. I’m sad to say T10 compliant object storage was a total flop. There are no object disk drives today and we are still dealing with block storage with no end in sight. FCoE had the promise of taking Fibre Channel out of the market, but up to this point FCoE has not lived up to market expectations.
iSCSI, however, started slowly about 11 years ago and up until recently there were few iSCSI targets except low-end storage, but today both mid-range and enterprise storage use iSCSI. I even use iSCSI as a NAS target at my home, because it was easy to setup and was significantly faster than using it as a network drive. Now my online Internet backup service works with an iSCSI target, but not with a network drive.
FCoE vs. iSCSI
In this article I’ll compare and contrast FCoE and iSCSI based on research I did as well as research and insight from some of my colleagues. That is, we’ll explore the advantages and disadvantages of both FCoE and iSCSI. Of course, as is often the case, I’ll take a different tact than many of my peers.
Also, one note of caution: There are all kinds of benchmarks and performance tests available on both sides of this issue. I have reviewed many of them. Some of the things I have seen are benchmarks comparing 4Gb Fibre channel to 10Gb Ethernet years after 8Gb FC was released. I’ve also seen research using drivers or other parts of the software stack that are specifically tuned to prove a point. And I’ve seen tests that didn’t use known features and functions that are generally available. My approach is to assume that you have efficient software and an optimized hardware environment and to not implement some SBT (slimy benchmark tricks). Remember I was a benchmarker.
Baseline Facts
I want to review two important areas that I think have been left out of discussions that I have seen:
- IP header size
- Real file system issues not streaming data or IOs per second (IOPS)
IP Header Size
As we all know, the “I” in iSCSI stands for IP. The IPv4 header size is 20 octets or 20 bytes. The world is moving from IPv4 to IPv6. Now there are many who might say that in the future we will use IPv6 between machines across the WAN, but IPv4 locally. I know of a number of companies — as well the U.S.government and foreign governments — that say if you refer to the “I word” it will be IPv6. This, of course, is not true today, but it will become more true later this year and in 2012 and 2013. What this means is going from 20 bytes to 40 bytes. The point here is that FCoE does not require this extra 20 or 40 bytes for IP encapsulation. Remember that more than likely this will become 40 bytes given the IPv6 requirements coming.
Real File System Issues
Whether we like it or not, testing any I/O performance in the real world must take into account file systems, and some file systems are more efficient than others. Measuring IOPS, streaming or any other test to just a LUN does not provide a realistic comparison to real I/O operations to a file system. I and others have pointed out for years that the bottleneck for most modern file systems is I/O for file system metadata. I will define metadata to include the following:
- File system superblock, which is read at mount time to understand the file system topology and needs to be updated occasionally for some file systems.
- File system allocation maps are updated regularly with files being written and deleted. Maps can be represented a variety of ways including bitmap and btrees.
- File system inodes and extended attributes, which include the location of the file, the attributes for the file such as UID, GID, access time, creation time and many other attributes, some of which are file system dependent
My premise is that metadata performance is often the limiting factor to file system performance. The metadata bottleneck is well-known and is getting worse given the massive growth in counts of files. Updating metadata is almost always a small block random I/O problem and, in some cases, must be done synchronously to meet various POSIX and file system requirements.
Why Are These Things Important?
Understanding that iSCSI requires a header is important as it adds space on the channel. Like it or not space is important.
Request Size in bytes | % of Head space IPv4 | % of Head space IPv6 |
512 | 3.91% | 7.81% |
1024 | 1.95% | 3.91% |
2048 | 0.98% | 1.95% |
4096 | 0.49% | 0.98% |
8192 | 0.24% | 0.49% |
1684 | 0.12% | 0.24% |
Depending on the file system, I am aware of request sizes from 512 bytes all the way up to 16384 bytes and, in some cases, greater (as the table above details). Admittedly, 512 byte requests are not that common. Using 7.81 percent of the bandwidth of the channel with just IPv6 header information is not going to help efficiency nor is it going to help latency as more data must be transmitted given the header requirement. FCoE does not have an additional IP header and does not suffer this channel limiting overhead. The overhead of file system metadata must be considered when evaluating FCoE vs. iSCSI. Maybe there are some tests that do this, but I have not seen any across a number of file systems. So I asked myself this question: If FCoE seemingly provides less overhead, than why is iSCSI making bigger inroads today than FCoE.
The Reasons Why
As stated, iSCSI has been around for more than a decade and got off to a very slow start, but during the past decade there have been many developers who have become familiar with iSCSI and many products have been brought to market. Also, during this 10-year period, vendors have learned a great deal about how to optimize the implementation of the protocol on both the host and target. FCoE, on the other hand, is a relatively new protocol with far fewer years of optimization behind it. That does not make iSCSI inherently better.
I believe iSCSI will continue to dominate the market even though it should not from a performance point of view. The fragmentation of the FCoE market from vendors is not helping matters at all and if the vendors keep moving towards products that do not completely interoperate — I mean completely interoperate like Fibre Channel does today — then FCoE will not be successful in the market.
It never ceases to amaze me that storage vendors often shoot themselves in the foot and, in this case, the head in terms of performance in the name of a perceived market advantage. I really don’t get it. Storage performance is critical to the survival of the storage industry and every little bit of performance, especially in a critical area such as metadata performance, is limiting file system scaling.
I have been ranting for years that the industry is doing nothing to address the lack of scaling for storage performance. We have narrow interfaces between the application, operating system, file system and storage. The few times the industry got together to make dramatic changes such as OSD, it was not widely adopted. Vendors often tell me that the reason OSD did not make it was due to the economic downturn. Well, we are coming out of the economic downturn and I hear nothing about OSD products coming to market.
My fear is that FCoE will suffer the same fate as OSD. The storage vendors seemingly race to the mediocrity. This has happened with many industries in the U.S. and the results have not been pretty for the country. If the storage industry does not make some dramatic changes I think that tiers 0 and 1 might become things of the past and we will only have nonvolatile storage on servers and archival storage — not a good situation or maybe that is what vendors want.