51–60 of 406 posts
I recall using a device called the "Mini Maxwell" by this company: https://www.iwl.com/ It enabled you to simulate network slow-downs, packet-loss, packet corruption, packet reordering and more. It was so critical in testing our highly network sensitive software.
Having a lot of experience commuting on underground public transport (intermittent, congested), and living/working in Australia (remote), I can safely say that most services are terrible for people without "ideal" network conditions. On the London Underground it's particularly noticeable that most apps are terrible at handling network that comes and goes every ~2 minutes (between stops), and which takes ~15s to conne…
> In Australia you're just 200ms from everything most of the time. (...) > The only app that I am always impressed with is WhatsApp. It's always the first app to start working after a reconnect, the last to get any traffic through before a disconnect, and even with the latency, calls feel pretty fast. The 200ms is telling. I bet that WhatsApp is one of the rare services you use which actually deployed servers to Aust…
So, I used to work at WhatsApp. And we got this kind of praise when we only had servers in Reston, Virginia (not at aws us-east1, but in the same neighborhood). Nowadays, Facebook is most likely terminating connections in Australia, but messaging most likely goes through another continent. Calling within Australia should stay local though (either p2p or through a nearby relay).
There's lots of things WhatsApp does to improve experience on low quality networks that other services don't (even when we worked in the same buildings and told them they should consider things!)
In no particular order:
0) offline first, phone is the source of truth, although there's multi-device now. You don't need to be online to read messages you have, or to write messages to be sent whenever you're online. Email used to work like this for everyone; and it was no big deal to grab mail once in a while, read it and reply, and then send in a batch. Online messaging is great, if you can, but for things like being on a commuter train where connectivity ebbs and flows, it's nice to pick up messages when you can.
a) hardcode fallback ips for when DNS doesn't work (not if)
b) setup "0rtt" fast resume, so you can start getting messages on the second round trip. This is part of noise pipes or whatever they're called, and tls 1.3
c) do reasonable-ish things to work with MTU. In the old days, FreeBSD reflected the client MSS back to it, which helps when there's a tunnel like PPPoE and it only modifies outgoing syns and not incoming syn+ack. Linux never did that, and afaik, FreeBSD took it out. Behind Facebook infrastructure, they just hardcode the mss for i think 1480 MTU (you can/should check with tcpdump). I did some limited testing, and really the best results come from monitoring for /24's with bad behavior (it's pretty easy, if you look for it --- never got any large packets and packet gaps are a multiple of MSS - space for tcp timestamps) and then sending back client - 20 to those; you could also just always send back client - 20. I think Android finally started doing pMTUD blackhole detection stuff a couple years back, Apple has been doing it really well for longer. Path MTU Discovery is still an issue, and anything you can do to make it happier is good.
d) connect in the background to exchange messages when possible. Don't post notifications unless the message content is on the device. Don't be one of those apps that can only load messsages from the network when the app is in the foreground, because the user might not have connectivity then
e) prioritize messages over telemetry. Don't measure everything, only measure things when you know what you'll do with the numbers. Everybody hates telemetry, but it can be super useful as a developer. But if you've got giant telemetry packs to upload, that's bad by itself, and if you do them before you get messages in and out, you're failing the user.
f) pay attention to how big things are on the wire. Not everything needs to get shrunk as much as possible, but login needs to be very tight, and message sending should be too. IMHO, http and json and xml are too bulky for those, but are ok for multimedia because the payload is big so framing doesn't matter as much, and they're ok for low volume services because they're low volume.
Earlier quoted context omitted.
Or maybe, just get rid of the firewall. I am all for nimble tech, but enabling the Chinese government is not very high on my to-do list.
Whether you like it or not, over 15% of the world's population lives in China.
[1] Local-first software: You own your data, in spite of the cloud:
https://www.inkandswitch.com/local-first/
[2] Local-first Software:
[3] Leonard Kleinrock: Mr. Internet:
https://www.latimes.com/opinion/la-oe-morrison-use24-2009oct...
Earlier quoted context omitted.
Whether you like it or not, over 15% of the world's population lives in China.
I guess they should fix their government, then.
The Braid Protocol allows multiple synchronization algorithms to interoperate over a common network protocol, which any synchronizer's network messages can be translated into.. The current Braid specification extends HTTP with two dimensions of synchronization:
Level 0: Today's HTTP
Level 1: Subscriptions with Push Updates
Level 2: P2P Consistency (Patches, Versions, Merges)
Even though today's synchronizers use different protocols, their network messages convey the same types of information: versions in time, locations in space, and patches to regions of space across spans of time. The composition of any set of patches forms a mathematical structure called a braid—the forks, mergers, and re-orderings of space over time.
I still think engineering for slow internet is really important, and massively under appreciated by most software developers, but ... LEO systems (like Starlink, especially StarLink) essentially solve the core problems now. I did an Arctic transit (Alaska to Norway) in September and October of 2023, and we could make FaceTime video calls from the ship, way above the Arctic Circle, despite cloud cover, being quite far…
Which I thought ties into the discussion about slow internet nicely.