Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
The United States leads in storage now, but what about in the future? I am not, nor have I ever been, xenophobic, but I have been wondering why most of the major storage developments related to filesystems, channels, RAID controllers, disk drives, tape drives, standards and so on take place in the United States, and not somewhere else.
In this article, I will explore why things are the way they are, and whether they are likely to continue on the same path.
Some Reasons Why
Here are the reasons I came up with:
- Historically, most hardware development has taken place in the United States and with servers goes storage
- Storage requirements in the United States exceed requirements elsewhere; the market is driven by requirements, and the rest of the world is behind our requirements
- For years, the U.S. Government (e.g., National Science Foundation and Defense Advanced Research Projects Agency) has funded universities in the United States to do basic research on storage and build filesystems
- A combination of all of the above
Alternatively, someone could say none of the above, which I will totally disagree with, but I am open to a reasoned opinion.
The United States, like it or not, has been the center of development of most servers and PCs. They might not have all been made here, but most of the development work was done here. This holds true for the mainframes from IBM, Univac, CDC and others in the 1950s through 1970s, to RISC servers in the later 1980s through today, and, of course, the PC. Each of these systems required storage systems to support the computation.https://o1.qnsr.com/log/p.gif?;n=203;c=204650394;s=9477;x=7936;f=201801171506010;u=j;z=TIMESTAMP;a=20392931;e=i
This is not to say that other areas of the world, most notably Japan, did not develop storage and technology. Hitachi, Fujitsu and NEC all had computer systems during much of that period that were products of their own development. Fujitsu had a disk and tape group, and Toshiba had disk drive development and purchased the disk group from Fujitsu a few years ago. Hitachi, of course, had a disk drive group, but much of the innovation came from the purchase of IBM's disk drive group, and Hitachi is now being integrated into Western Digital's portfolio.
Disk drives alone are not a storage ecosystem in my opinion. As of this writing, Fujitsu, Hitachi and Toshiba have not been leading disk technology vendors in terms of innovation. Each might have led for a short period of time in a small area, but none have been market leaders over the long term.
The way I see it, the nation with the largest number of hardware vendors in the past 40 years has tried to develop some storage technology, but it has never been successful in the long run. Nor has it developed an end-to-end ecosystem with filesystems, channels and storage.
What About Requirements?
Does the United States lead the world in storage because we have the requirements that require innovation? Requirements in my opinion can take the form of software developed that has needs that lead to storage innovation (e.g., databases needing fast file I/O or higher reliability or something). Consider the requirement for larger storage volumes for databases and other needs and the development of VxVM from Veritas for volume management and VxFS as a file system in the early 1990s. This was ported to many different operating systems and supported by many vendors. Prior to this was the PC explosion, and that prompted the development of SCSI standard and SCSI disks and adaptors. In the 1990s, RAID development was occurring. That lead to the explosion of RAID devices from many vendors, mostly because we needed larger, more reliable and faster storage for databases. The explosion of email and the web contributed to this.
Did the Internet and databases spur the development of storage technology? Did the database requirements from the multiple vendors in the United States create an environment where storage development was required for the applications? Is the United States' standing as the largest market for technology and development go hand and hand with technology?
What About Research?
I was not involved in research in the 1950s and 1960s or even 1970s, but from the information I have read about the period, the United States provided funding to research many areas including, but not limited to, CPUs, memory, channels, filesystems and storage--in other words, pretty much everything in the computing ecosystem. Money was spent at many universities around the United States, which resulted in the development of many game-changing technologies, from the Internet to filesystems. I have no idea how much was spent, but the computing industry would not have been the same today without the innovation spurred by the funding given for research at many many universities around the nation. The research was transformative, and many companies were formed as an outgrowth of the research. Funding came from various DoD organizations, NASA, NSF and others.
Of course, some of the money was wasted, but some things were developed that became everyday technologies. Not every invention can evolve into the Internet, or disk read/write heads, or Ethernet and so on. Basic research was the function of the U.S. government, and it allowed us to create many of the things we have today. Also note that much of the research money and many of the developments had to do with the goal of the U.S. government to win the "Cold War."
Is It a Combination of Factors?
After looking at the types of inventions, where we are today and where we have come from, I strongly believe the reason is this combination of factors. Whether it is politically correct today or not, the basic research funded by the U.S. government in the whole computing ecosystem, especially storage, is why in my opinion the United States has dominated the storage market since the beginning of the computer era. Although there are occasional technology advances from other countries in a specific area, no nation has developed a set of technologies that compete end-to-end with the storage infrastructure we have today. A nation might develop a small part of the solution set--for example, Japan develops most of the tape media technology--but no nation develops most of the end-to-end technology besides the United States.
Developing storage technology I think requires an ecosystem of technology, requirements and people who have a vision and have the freedom to innovate. I also think that funding research provides the basis of much of this innovation. Given the economic situation during the past few years, the storage industry has not seen much new technology coming out from the storage vendors. For the most part, the U.S. government has not spent significant funding on storage research in about a decade, but it has funded other areas, such as FPGAs, memory and other computational technologies. I am not sure that without research funding the lead and the pace of innovation that has occurred in the United States is sustainable; nor am I sure that another nation has the wherewithal to address the end-to-end storage stack.
So where does that leave us? I am beginning to believe we are left in storage limbo, where what is new is not significant innovation in the end-to-end development of storage. Progress does not happen just because of requirements. You need the people and the tools, ideas and money.
The question is, will another nation pick the ball up and run with storage innovation? My opinion is that if a nation wants to take the reins away from the United States in the area of storage innovation, it will take years of investment to develop the people with the right skills and the national infrastructure.
Right now, I do not see anyone standing up and expressing willingness to take the lead, but by the same token the United States is not standing up and leading the way it once did. Hence, storage limbo.
Henry Newman is CEO and CTO of Instrumental Inc. and has worked in HPC and large storage environments for 29 years. The outspoken Mr. Newman initially went to school to become a diplomat, but was firmly told during his first year that he might be better suited for a career that didn't require diplomatic skills. Diplomacy's loss was HPC's gain.