We need a replacement for TCP in the datacenter [pdf]

My son did work in graduate school for a clean-slate network implementation of a network for the datacenter. Maybe Google, I don't remember.

One issue I remember they addressed was, scheduling bandwidth for VM migration within their datacenter cloud. See, some customer reserves a 'machine' for their services but really they get something like a VM slice of a ginormous machine (Multiple TB memory, 100 cores or whatnot). Each customer gets some of that and thinks it's a machine of their own.

That customer slice shares the larger machine with maybe 10-100 other customers. Then somebody's slice starts to use more resources and has to be moved to a machine with more 'room'. That wants to be fast and seamless. It can be maybe 1TB of stuff. Their slice doesn't want to be interrupted for long. So this machine needs bandwidth that isn't subscribed for the migration. So does the target machine. So does the cloud network. Then all the addresses have to be re-homed.

Another issue: those competing slices need a virtual network adapter. They each think they own one (each is running a copy of linux or whatnot), but it has to be a physically shared and rationed device. All while using the TCP abstraction on a network-adapter abstraction on a driver abstraction, but really on their new network hardware that's actually present on the ginormous machine. This includes all the TCP features plus the bandwidth reservations the cloud needs etc.

So yes it's abundantly obvious that the datacenter needs (has) a new network.

Because the underlay TCP is lossless (being TCP), every time the overlay TCP has to retransmit, it adds to the queue of things that the underlay TCP has to retransmit (and the need to retransmit happens more or less at the same time).

So instead of linear increase in the number of packets, you get ~quadratic.

This balloons the required throughput needed to “rectify” the issue from the protocol standpoint at both levels - usually precisely at the point when there’s not enough capacity in the first place (the packet loss is supposed to signal congestion).

If you are very lucky, the link recovers fast enough that this ballooning is small enough to be absorbed by the newly available capacity.

If the outage is long enough, the rate of build-up of retransmits exceeds the capacity of the network to send them out - so it never recovers.

Needless to say, the issue is worse with large window in overlay TCP session - e.g. a sudden connectivity blip in the middle of the file transfer.