Engineering for Slow Internet

Summary: since 2024-05-06, users of all networks also get LTE in the U-Bahn thanks to a project between BVG and Teléfonica (not surprising that Teléfonica deployed the infra because they had the best U-Bahn LTE coverage beforehand)

So Google Fiber needed a pure JS Speedtest for installers to verify connections. Installers were issued with Chromebooks, which don't support Flash and the Ookla Speedtest at the time used Flash. There's actually good reasons for this.

It turns out figuring out the maximum capacity of a network link is a nontrivial problem. You can crash the browser with too much traffic (or just slow down your reported result). You can easily under-report speed by not sending enough traffic. You have to weigh packet sizes with throughput. You need to stop browsers trying to be helpful by caching things (by ignoring caching headers). There's a long list.

So I did get a pure JS Speedtest that could actually run up to about ~8.5Gbps on a 10GbE link to a Macbook (external 10GbE controller over TB3).

You learn just how super-sensitive throughput is to latency due to TCP throttling. This is a known and longstanding problem, which is why Google invested in newer congestion control schemes like BRR [1]. Anyway, adding 100ms of latency to a 1GbE connection would drop the measured throughput from ~920-930Mbps to a fraction of that. It's been a few years so I don't remember the exact numbers but even with adjustments I recall the drop off being like 50-90%.

The author here talks about satellite Internet to Antarctica that isn't always available. That is indeed a cool application but you don't need to go this extreme. You have this throughput problem even in Australia because pure distance pretty much gives you 100ms minimum latency in some parts and there's literaly nothing you can do about it.

It's actually amazing how much breaks or just plain sucks on that kind of latency. Networked applications are clearly not designed for this and have never been tested on it. This is a general problem with apps: some have never been tested in non-perfect Internet conditions. Just th eother day I was using one of the Citi-bike apps and it could hang trying to do some TCP query and you'd every now and again get "Connection timed out" pop ups to the user.

That should never happen. This is the lazy dev's way of just giving up, of catching an exception and fatalling. I wish more people would actually test their experience when there was 100ms latency or if there was just random 2% packet loss. Standard TCP congestion control simply doesn't handle packet loss in a way that's desirable or appropriate to modern network conditions.

[1]: https://cloud.google.com/blog/products/networking/tcp-bbr-co...

For iPad, I think it's like the sibling notes; expected use is very low, so it didm't justify the engineering cost while I was there. But I see some signs it might happen eventually [1]; WhatsApp for Android Tablets wasn't a thing when I was there either, but it is now.

For the four device limit, there's a few things going on IMHO. Synchronization is hard and the more devices are playing, the harder it is. Independent devices makes it easier in some ways because the user devices don't have to be online together to communicate (like when whatsapp web was essentially a remote control for your phone), but it does mean that all of your communications partner's devices have to work harder and the servers have to work harder, too.

Four deviced covers your phone, a desktop at home and work, and a laptop; but really most of the users only have a phone. Allowing more devices makes it more likely that you'll lose track of one or not use it for long enough that it's lost sync, etc.

WhatsApp has usually focused on product features that benefit the most users, and more than 4 devices isn't going to benefit many people, and 4 is plenty for internal use (phone, prod build, dev build, home computer). I'm sure they've got metrics of how many devices are used, and if there's a lot of 4 device users and enough requests, it's a #define somewhere.

[1] https://www.macworld.com/article/668638/how-to-get-whatsapp-...