Distributed Systems Classics (2017)

WhatsApp backend in Erlang + used in telecoms

Not OP, but a good resource for distributed consensus specifically is Tim Roughgarden's YouTube playlist "Foundations of Blockchains" [0] (hear me out, despite the title - see below). It's 85 videos over 12 lectures, rigorous, very well explained (but assumes some CS fundamentals). Not original work, but gives the conceptual framework and background so that one can read these classic papers in context.

Per-lecture reading lists are on the course page [1]. He also points at Elaine Shi's Foundations of Distributed Consensus and Blockchains [2] and Andrew Lewis-Pye's Consensus in 50 pages [2] as background, though these are more textbooks, not papers.

Remarkably, the first 7 lectures deal with permissioned systems, recapitulating the classic results of consensus in distributed systems (Dolev Strong, FLP impossibility, CAP) - no blockchain in sight. This takes us to the state of the art at the end of the 1990's (with algorithms that can achieve consensus in the presence of byzantine failures, namely Byzantine Paxos and PBFT, though he discusses a modern variant, permissioned Tendermint from 2014).

Lecture 8 stays permissioned and proves consistency and chain quality for the longest-chain rule. Only at Lecture 9, with proof of work, does anything specifically blockchain appear; then L10 block rewards and selfish mining, L11 transaction fee mechanism design; L12 proof-of-stake sybil resistance with the whole litany of attacks possible there.

Very good series in my view, and shows how little technical merit this whole blockchain circus has - nearly all the great properties people tout (reliability, consistency, audibility, availability) can be achieved with good old permissioned tech more efficiently, with pretty instant and deterministic finality.

Anyway, I found the series worth watching for the classical consensus material alone.

[0] https://www.youtube.com/playlist?list=PLEGCF-WLh2RLOHv_xUGLq...

[1] https://timroughgarden.github.io/fob21/

[2] https://elaineshi.com/docs/blockchain-book.pdf

[3] https://lewis-pye.com/2022/08/15/consensus-in-50-pages/

I am a Lamport admirer.

I gradually realized that Lamport is more of the godfather of distributed system than Hinton is to deep learning. Lamport is less prominent than Shannon is to information theory. Shannon is the closest to any title of "gold-like" figure to a scientific discipline of universal relevance in modern society.

Lamport specifically revealed a philosophical connections between computer systems and physics, in the parallel of distributed consensus to relativity theory. To me, the enlightenment is that, the relations between events happening in a distributed system, is more fundamental than their absolute ordering, thus the central role of an "observer". I haven't really analyzed if this realization is from Lamport's papers, or my general reading and thinking, but I am moderately confident that general readings are heavily influenced by Lamport's papers, or can be traced back to be compatible with Lamport's thinking. I have not seriously study if this connection is valid in depth, which might be another pure amateur speculation of mine.

One thing I think Lamport falls short is that his writing is not easy to read and understand. I unconciously feel that Lamport (and Dario from Anthropic) probably share a hidden sense of intellectual supriority grew from their own experience throughout their career. So their writing (and Dario's gospel) all share a unchangable sense of narration from their own delicate and graceful ideas, much less of faciliating the understanding to their audience. In this cateogry, Shannon is abosolutely superior in any measure, in his writing, ideas are so naturally presented, although the implications of the ideas remain elusive due to the inherent depth.

Also, among the 3 prominent figures of modern AI: Hinton/Bengio are more like Shannon, Lecun is closer to Lamport.

Enough random rambling. Lamport, as indicated by the outweight presence in this list, is no doubt the single most important scientist in distributed systems.

There's still a lot of WhatsApp Erlang tooling being published, so I'm pretty sure it's mostly still Erlang. I think I still know a couple people who would share with me if it was rewritten, although I left in 2019.

Erlang is simply the right language to build a chat server with, and it doesn't make sense to rewrite it in another language, even though Facebook did that with the server for Messenger. :P

Certainly parts of the WhatsApp stack have changed considerably. No more FreeBSD / no more only a handful of OS processes per node, much less Mnesia (not sure if there's any Mnesia left... I don't get that level of information). I've seen signs that they may have returned to large nodes, so they might be back to millions of connections per node, but that was very much not the case when I left; connection counts had dropped on our FreeBSD nodes because we were doing a lot more on the chat channel and then again because the Facebook nodes we migrated to were so much smaller than our FreeBSD nodes.

A favorite Knuth story to illustrate: He was re-writing the 8080 code for the processor that ran our new Alphatype CRS Phototypesetter, since the manufacturer's code wasn't suitable. Real-time, 7 levels of simultaneous interrupt stuff, since it controlled two stepper motors that moved the lens around the photographic paper, plus four custom font outline-tracing S100 cards that flashed vertical segments of characters at just the right moment on the CRT that the lens then focused onto the paper. No debugability other than a 4x4 keyboard and a 15(?) character display. I was responsible for the code on the other end of the RS232 port that connected it to the DEC mainframe, sending down character outlines on the fly to the very limited memory on the 8080, along with the properly-sorted character placement info ultimately from TeX.

Anyway, we'd been doing morning, noon, and night debugging sessions together for a number of days. On maybe the 3rd morning, I arrived at his office and was speaking with his secretary of many years, Phyllis Winkler, when Knuth came in. He handed her a stack of maybe a dozen or two pages, saying "Here's a paper I wrote; please type it in." When? When did he write it? It's still a mystery to me.