Download the authoritative guide: Enterprise Data Storage 2018: Optimizing Your Storage Infrastructure
The obvious solution for network storage is a centralized NAS for either NFS access or CIFS access or even both. This would be a centralized storage server with a fair amount of capacity and/or a fair amount of performance. There can be a large number of drives, perhaps an accelerator module or two, and some other "magic" to improve performance since microservers can be packed so densely.
On the client side, let's assume that each microserver has a Gigabit Ethernet (GigE) link to a network to which the NAS server connects. Let's also assume the server doesn't use the local disk for storage or that the node is basically diskless.
Even if we play "best case" and assume maybe one out of ten servers is using the NAS storage, we still need 28.8 10GigE links, or 7.2 40GigE links, or 5.2 FDR IB links. This is still quite a bit of NAS performance.
Because of the extreme densities that microservers offer, you could need an uber-powerful NAS storage solution for every rack. Obviously this is an untenable situation. Or is it?
NAS In The Rack Option (NITRO)
What is needed for microservers is something in-between local storage and a big, bad, centralized NAS (BBC-NAS). I call it NITRO (NAS In The Rack Option). The idea is to push a reasonably sized NAS into the rack with a subset of the microservers rather than have a BCC-NAS for everything. This storage is intended to be a fast-scratch or working space storage solution that is not necessarily backed up on a regular schedule.
This solution allows a certain number of microservers to have their "own" NAS box to use for network storage. Then, periodically, the data is copied or "rsync-ed" to the BBC-NAS (Big, Bad, Central NAS).
I would like to lay claim to this idea, but all I can lay claim to is the acronym. In my day job, I have seen a few customers implement this idea with great success. These customers have shown that this is definitely doable — if you carefully plan the NITRO solutions, how you manage them, and how data moves to and from them.
NITRO effectively creates two layers of network storage. The first "NITRO" layer, the one highest up the storage pyramid, provides NAS storage that is close to the servers. The second layer of NAS, the BBC-NAS, is further down the storage pyramid and doesn't have to have the performance that would necessary if it were catering to the entire rack. Sounds pretty reasonable, so let's go a step further and examine what a NITRO would look like.
Recall that there are microservers that can fit up to 288 servers into a single 4U chassis (probably diskless). Again assuming that each server has a single GigE, then you have 288 GigE lines coming from the servers. In the worst case of all servers in a single 4U chassis communicating to a single NITRO at the same time, this is the same as 28.8 10GigE lines, or 7.2 40GigE lines, or 5.14 FDR IB lines. If we go back to assuming that only 1/10 of those servers are communicating at one time, the NITRO only needs 28.8 GigE lines, or 2.88 10GigE lines, or 0.72 40GigE lines, or 0.51 FDR IB lines. This configuration sounds very reasonable for a NAS solution.
In the case where each server has some local storage, then there are perhaps 96 servers in 4U. If each server has a single GigE line, then in the worst case with all servers communicating at the same time, you have 96 Gbps going to the NAS gateway or about 10x 10GigE lines (2.5x 40GigE). If we use the 10:1 ratio, this changes to one 10GigE and 40GigE becomes overkill. Again, a very reasonable configuration for a NAS solution.