Cloudflare Meerkat - Globally distributed consensus

Illustrated with meercats looking in different directions.

There’s a weird yet common attitude that nobody has to implement operating systems, databases, compilers, cryptography, consensus algorithms, etc because you should use off the shelf solutions.

This is obviously true, and there are surely more web devs than, say, compiler engineers.

But then the logic seems to go as far as to imply nobody does or should be doing these things. Where do they think the “off the shelf” solutions came from?

I worked at , and many critical systems written in the 90’s and early 2000s used bespoke compilers, caches, and consensus protocols that were hand rolled by the original developers.

Back then, most of these ideas were already present in academia. But the industry tech wasn’t quite there yet to run these huge services at scale without running into problems (paying a vendor, licensing, the open source solution not quite there or tunable enough to easily integrate it at the needed scale, no company wide consensus on a shared approach yet).

A lot of these handwritten implementations eventually informed the “off the shelf industry solutions” we have today.

I think that’s what gets me down about tech these days. There’s a weird almost anti innovation attitude, which is the opposite of what made me fall in love with this profession. I’m not saying to hand roll a consensus algorithm at your next start up. But there’s definitely a vibe these days that any sort of theoretical, creative, or innovative thinking is suspect. Get back to selling ads!

I feel like it would be much better if the article focused on QuePaxa because IMHO it's an algorithm that finally brings some novel ideas to concensus (e.g. not relying on timeouts) by kinda coming at it from a gossip protocol angle and is not getting the attention it deserves. The post shouldn't have focused and introduced Meerkat which hasn't been fully developed and tried in production. If they clearly presented the pros and cons vs not just Raft (which is popular but doesn't even play in the same league because it is relies on a leader) but other leaderless or multi-leader concensus protocols that would have been of greater value. The Paxos family of algorithms are a much closer fit here and there's a reason why some serious large planet scale systems choose it over Raft.

E.g. 1. Intro about issues with concensus 2. Intro to QuePaxa 3. Comparison to other algos that are close to it 4. Mentioning active work on implementation via Meerkat and intent to bring to production with followup posts.

As always when it comes to concensus it's all about trade-offs. And with QuePaxa that might be the increase in messages (note: I don't mean message round-trips). We'll see how it goes but it will definitely be interesting.

The article lets itself down when it reveals it's had some marketing "polish" added: "where certain resources (like an AI model instance) are stored"

They could have picked something far more concrete (for Cloudflare's business and the type of services they deploy) than "AI model" there, which is clearly picked just to make it seem 'current' and on-trend.

I imagine something akin to a bat signal that alerts Aphyr to these types of posts. Looking forward to the Jepsen write-up.