I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."
“…but I repeat myself.”
Why use Paxos instead of Raft?
21–30 of 46 posts
Re: Why use Paxos instead of Raft?
#22The author is here, happy to answer questions if any.
Besides storage itself, the Postgres compute layer has a good amount of (transient) state that doesn't lend itself to either compute nodes or clients springing in and out of existence in a serverless environment. For instance, a fresh compute node with an unfilled cache can perform horribly, and Postgres client connections don't scale well with transient clients. Both of these problems, and others in the same category, were very noticeable for Aurora Serverless. My understanding is that AWS mitigates these two by an elaborate cache-filling service for new nodes, and a pgbouncer-style proxy pooling connections and hiding compute nodes being rescheduled from clients.
What's Neon's point of view about transient state in nodes? Is there a world where serverless client connections are stateless, or is the set up overhead not expected to be worth the cost?
Re: Why use Paxos instead of Raft?
#23The author is here, happy to answer questions if any.
Do you attempt to guarantee linearizability of read-only operations? The scenario I'm concerned about is when a partitioned compute node is processing a read-only transaction from a partitioned client, and neither has noticed the partitioned compute node has been replaced in a later term. Do you use a lease system for this that relies on the partitioned compute nodes to be able to accurately measure the passaged of t…
Good catch! Currently, we don't, and we rely on k8s to stop the old node. Technically speaking, if k8s and our control plane are always good at stopping the old primary, we don't need consensus at all. So that is more of a question of what set of problems we can see if there is a bug in our orchestration code. Split-brain seemed to be unacceptable. But with stale reads, we decided that we can only rely on k8s without double-checking that on our side.
Re: Why use Paxos instead of Raft?
#24The author is here, happy to answer questions if any.
Besides storage itself, the Postgres compute layer has a good amount of (transient) state that doesn't lend itself to either compute nodes or clients springing in and out of existence in a serverless environment. For instance, a fresh compute node with an unfilled cache can perform horribly, and Postgres client connections don't scale well with transient clients. Both of these problems, and others in the same categor…
This is a very good question. We are working on it and will be publishing a blog post on autoscaling very soon. We are experimenting with VM Migration technology that would allow to transfer the state between compute nodes and failover traffic.
We have some encouraging early results, but haven't committed to a particular technology (like cloud hypervisor) yet.
Re: Why use Paxos instead of Raft?
#25So would using Neon negate the need for something like Citus for scaling out a postgres database?
It's a different approach, planetscale and Citus are sharding that is intended to be mostly transparent. It's not 100% transparent, but both get pretty close. Neon is more of an aurora approach detaching storage from the compute, you could scale up to more replicas and it could enable other functionality, though Postgres already can handle a pretty high replica count so you can scale out reads that way.
That's right. The important observation is that in OLTP queries are mostly small and can fit into one node. Neon architecture allows to scale read throughput by spinning up read replicas (or read endpoints to the same storage in our case).
Citus (Cockroachdb, Yugabyte) has distributed compute which allows to engage multiple nodes per queries. This helps with analytical queries AND with scaling writes. But you lose out on compatibility and predictability of performance. Shared nothing systems are no longer Postgres.
Re: Why use Paxos instead of Raft?
#26I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."
Assuming we're thinking of the same person, I believe it went:
"There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."
Re: Why use Paxos instead of Raft?
#27The author is here, happy to answer questions if any.
Besides storage itself, the Postgres compute layer has a good amount of (transient) state that doesn't lend itself to either compute nodes or clients springing in and out of existence in a serverless environment. For instance, a fresh compute node with an unfilled cache can perform horribly, and Postgres client connections don't scale well with transient clients. Both of these problems, and others in the same categor…
Right. Our design guideline is to get as much serverless behavior as possible while keeping full Postgres compatibility (in terms of features and expected performance). Single node Postgres can give you hundreds of thousands of small RW queries per second, so competing connections should be a few compare-and-swap instructions away from the shared state to provide this performance. So for the primary, it means it should be just a Postgres in the container or VM, and we have to deal with consequences (cache pre-warm, handle cross-node migrations, etc).
However, read-only nodes require less coordination, and we have way more freedom there, so read-only Postgres as a function seems to be a more feasible concept.
Re: Why use Paxos instead of Raft?
#28I'm curious how many nodes end up in the consensus group, presumably you don't want more than 3 because throughput scales 1/N, unless their implementation can alleviate that significantly.
Currently, we deploy three safekeepers, one in each AZ. We need to collect more stats on failure rates, and maybe we will go to 6 (3 AZs with two nodes in each) as Aurora does.
Re: Why use Paxos instead of Raft?
#29I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."
I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: https://youtu.be/QVvFVwyElLY?t=2370
Re: Why use Paxos instead of Raft?
#30The author is here, happy to answer questions if any.
What is Neon? What is Paxos? what is Raft?