Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why use Paxos instead of Raft?

neon.tech

Post reply on HN
Print view View original

31–40 of 46 posts

Re: Why use Paxos instead of Raft?

#31

Post by erichocean »

kelvich wrote: post #4

The author is here, happy to answer questions if any.

Thank you for publishing the TLA+ model, that dramatically increases my level of trust in Neon. (I'm on the early adopter list, got my invite a week or two ago but haven't been able to give it a spin yet.)

> Right now, such a change requires humans to be in the loop to ensure that the old safekeeper is actually down. It is on our roadmap to automate this procedure.

If you do implement this (which I don't recommend), be certain to also model it with TLA+. This level of automation, IMO, requires a human in the loop + a ton of visibility tracking on when it is happening.

A good way to roll it out is "semi-automation"—implement the automation but use it to ask a human to approve. The human will then do the normal (manual) verification. After you've run that successfully for a year, and your TLA+ model passes, you can then decide to fully automate without a human in the loop.

Otherwise, you're asking for an outage (caused by bad failover) IMO, and possibly data loss.

Re: Why use Paxos instead of Raft?

#32

Post by toomim »

mgdev wrote: post #6

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

For some reason, people considering PAXOS/RAFT don't tend to consider CRDT/OT synchronization. I think this is a big oversight.

We should start considering CRDT/OT/VCS/Diffsync approaches to distributed systems as well. They present a very nice alternative approach: whereas PAXOS/RAFT implement a consistent "distributed state machine", a CRDT, OT, VCS, or Diffsync system implements consistent "distributed state", upon which one can build a machine as a function of the state.

This latter approach is actually simpler, IMO, because it encapsulates all the challenge of distributed consistency within a smaller subset of the problem — state synchronization. This makes it more generally re-usable. When you create a system, you can just use an off-the-shelf library & algorithm to synchronize your data over a network, and then write synchronous functions on top of that to represent the system you want, however you want, without having to understand PAXOS/RAFT.

Re: Why use Paxos instead of Raft?

#33

Post by asadawadia »

nikita wrote: post #18
asadawadia wrote: post #13

Earlier quoted context omitted.

if storage and compute are separated - how is storage mounted on to the compute? Generally you can attach a volume only to one server at a time

We changed Postgres to send WAL to safekeepers and read from page servers: https://neon.tech/blog/architecture-decisions-in-neon/

Safekeeper to page servers take some time

what happens when the compute server issues a read for something that has made it to the WAL servers but not the page servers?

Re: Why use Paxos instead of Raft?

#34

Post by winkywooster »

ChadNauseam wrote: post #15
mgdev wrote: post #6

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

It's a fun quote, it reminds me of the "every sufficiently advanced program contains a bug-ridden implementation of half of common lisp", or something along those lines. But there really is a wide world of distributed consensus systems out there (although Paxos is easily the most elegant).

You mean “Greenspun's tenth rule”

https://en.m.wikipedia.org/wiki/Greenspun%27s_tenth_rule

Re: Why use Paxos instead of Raft?

#35

Post by difflens »

Great write up! I don't have much practical experience with Paxos/Raft other than coursework. I'm curious, if I wanted to insert a row into a table, what is the overhead of all these extra nodes in the insert operation now compared to a single table?

I realize the answer depends on how big the cluster is, what state it is in at any given moment etc, but I'm happy to accept back of the envelope calculations/estimations!

Re: Why use Paxos instead of Raft?

#36

Post by kelvich »

asadawadia wrote: post #33
nikita wrote: post #18

Earlier quoted context omitted.

We changed Postgres to send WAL to safekeepers and read from page servers: https://neon.tech/blog/architecture-decisions-in-neon/

Safekeeper to page servers take some time what happens when the compute server issues a read for something that has made it to the WAL servers but not the page servers?

Postgres tracks maximal LSN among the evicted pages and passes it to the pageserver in the page request. If the pageserver hasn't received that LSN, it will wait for it to arrive.

Re: Why use Paxos instead of Raft?

#37

Post by jsmith45 »

gamegoblin wrote: post #20
mgdev wrote: post #6

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

When I was at AWS I heard the same phrase from a DE, probably the same guy, and once heard him also say "Raft is just a special case of Paxos to try to simplify it, but regular Paxos isn't actually that hard, just use Paxos". I was fairly junior at the time, and Raft seemed much more approachable, but after really forcing myself to read and understand the Paxos paper, I see what he meant. I am pretty sure most of the…

https://arxiv.org/abs/2004.05074

This paper argues basically argues that raft has a different leadership election mechanism than paxos, but that if you tweak some terminology, and make a few relatively reasonable implementation choices for paxos they are otherwise pretty equivalent.

It even gives a raft style single page description of paxos (using raft style terminology), and shows how little it differs from the equivalent single page summary of raft.

The main implementation choices they use are: - combined roles into a single server role - enforce that log messages are decided in sequence (largely to avoid the having to specify the behavior of newly elected leader to propose operations for the gaps (possibly no-ops)) - numeric ballot number, rather than lexicographical pair (but this changes nothing except making the summary slightly easier to express)

Re: Why use Paxos instead of Raft?

#38

Post by qohen »

arthurcolle wrote: post #19
ChadNauseam wrote: post #15

Earlier quoted context omitted.

It's a fun quote, it reminds me of the "every sufficiently advanced program contains a bug-ridden implementation of half of common lisp", or something along those lines. But there really is a wide world of distributed consensus systems out there (although Paxos is easily the most elegant).

There's a variation of this on the subject of distributed systems re: Erlang/BEAM VM

Yes, from Robert Virding, one of Erlang's co-creators, from the Erlang mailing-list, from 2008 [0]:

After reading the blogs about how good Erlang's concurrency model is and how we just just made a super implementation of it in XXX I have been led to formulate Virding's First Rule of Programming:

Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

This is, of course, a mild travesty of Greenspun (*) but I think it is fundamental enough to be my first rule, not the tenth.

[0] http://erlang.org/pipermail/erlang-questions/2008-January/03...

Re: Why use Paxos instead of Raft?

#39

Post by dastbe »

rubiquity wrote: post #26
mgdev wrote: post #6

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

Assuming we're thinking of the same person, I believe it went: "There are three types of consistent distributed systems: paxos, broken protocols, and single points of failures."

i don't understand why people won't just say alv. he won't hurt you.

Re: Why use Paxos instead of Raft?

#40

Post by diroussel »

toomim wrote: post #32
mgdev wrote: post #6

I used to work in the orbit of a distinguished eng at AWS who was famous for saying something to the effect of, "At the bottom of any scaled distributed system is either Paxos, or a bug."

For some reason, people considering PAXOS/RAFT don't tend to consider CRDT/OT synchronization. I think this is a big oversight. We should start considering CRDT/OT/VCS/Diffsync approaches to distributed systems as well. They present a very nice alternative approach: whereas PAXOS/RAFT implement a consistent "distributed state machine", a CRDT, OT, VCS, or Diffsync system implements consistent "distributed state", upo…

Would this approach be resistant to a rogue actor. If one actor bad data would all the other actors still be able to reach consensus?

Paxos is complicated, but it’s well studied and proven.

Post reply on HN