UDP vs TCP

Introduction

Bufferbloat is the undesirable latency that comes from a router or other network equipment buffering too much data. It is a huge drag on Internet performance created, ironically, by previous attempts to make it work better. The one-sentence summary is “Bloated buffers lead to network-crippling latency spikes.”

The bad news is that bufferbloat is everywhere, in more devices and programs than you can shake a stick at. The good news is, bufferbloat is now, after 4 years of research, development and deployment, relatively easy to fix. See: fq_codel: wiki. The even better news is that fixing it may solve a lot of the service problems now addressed by bandwidth caps and metering, making the Internet faster and less expensive for both users and providers.

The introduction below to the problem is extremely old, and we recommend learning about bufferbloat via van jacobson’s fountain model instead. Although the traffic analogy is close to what actually happens… in the real world, you can’t evaporate the excessive cars on the road, which is what we actually do with systems like fq_codel wiki.

Still, onward to the blow.

[...]

You mean the networks in particular that you use? There's a lot of varied network equipment out there. Glenn wrote this article at least 8 years ago and for the types of games he works on (realtime 64-player online FPSes), I'm sure TCP still wouldn't work as well as a UDP solution.

> The trick is to hide the lag with animations

That's an understatement. A significant amount of work needs to be done with client-side prediction in order to give the appearance of playing in real time consistently with other players.

> I think google, facebook, and world of warcraft use tcp for their real time apps

Which real time apps are you referring to? In the context of this article, Glenn is talking about soft realtime systems (30 hz updates or more). There aren't any user-facing apps from facebook or google that I'm aware of that I would call "real time". For WoW, timing is less critical than an FPS and obviously TCP works well for them.

In modern days UDP is really only used in situations where you can afford packet loss and CPU is expensive, or when you need to bypass the built-in behavior of TCP on your OS. It's not as much about latency or data loss, usually UDP is used in places where processing power is expensive or load is too extreme to justify TCP.

Sometimes, like with DNS, it's better to just resend to request than burden a massively parallel server with maintaining TCP state and checksums.

The main issue I have with the above comment is that UDP absolutely does not guarantee better latency or have higher priority than TCP packets. The packets usually queue in the NIC buffers and downstream identically regardless of which protocol you use. The main draw of TCP besides data and ordering guarantee (which are CPU costly) is that (depending on your operating system) it follows some kind of automatic rate limiting algorithm be it TCP Vegas(Linux, delay based) or Reno (windows, loss based). These algorithms attempt to limit your connection "stream" to the line rate automatically and play somewhat nicely with each other on large scale.

In contrast, the main problem with using UDP is figuring out a smart way to rate the limit to avoid excessive packet loss. In some protocols like DNS anycast type things where only raw throughout matters, who cares if responses drop. Other times, like with LEDBAT or Microsoft's BITS, your main reason for using UDP is to roll your own rate limiting protocol, since otherwise you're stuck with what the OS gives you on TCP.

Anyone here have any experience using QUIC in any application of their own?

The custom congestion control makes me wonder if it only works alongside TCP traffic - once everything goes QUIC, then what happens? I looked for a bit about the story in ancient history of some blazing fast server OS TCP implementation that broke the rules so it fell over when more than one server was on the network, but couldn't find it.