Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Raft: Understandable Distributed Consensus (2014)

thesecretlivesofdata.com

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Raft: Understandable Distributed Consensus (2014)

#51

Post by benbjohnson »

mgenglder wrote: post #16
benbjohnson wrote: post #6

Author here. I'm happy to answer any questions although this project was from 10+ years ago so I could be a little rusty. Over the years I've been trying to find better ways to do this kind of visualization but for other CS topics. Moving to video is the most realistic option but using something like After Effects takes A LOT of time and energy for long-form visualizations. It also doesn't produce a readable output f…

This is wonderful. Can I ask how you created it? Stack used and sour e code? I'd love to create something like this to help visualize things I'm working with currently.

It's all done with d3 and JavaScript. The visualizations aren't deterministic so I ended up writing a shitty Raft implementation in JS. Overall it was a terrible approach because it was so time consuming but I made it work. You can find all the source code in this repo: https://github.com/benbjohnson/thesecretlivesofdata

Re: Raft: Understandable Distributed Consensus (2014)

#52

Post by benbjohnson »

huntaub wrote: post #32
benbjohnson wrote: post #6

Author here. I'm happy to answer any questions although this project was from 10+ years ago so I could be a little rusty. Over the years I've been trying to find better ways to do this kind of visualization but for other CS topics. Moving to video is the most realistic option but using something like After Effects takes A LOT of time and energy for long-form visualizations. It also doesn't produce a readable output f…

I just want you to know how much this visualization was appreciated. In my time working at AWS, I recommended this website to every one of our new hires to learn how distributed consensus works. Know that this has taught probably 50+ people. Thank you for what you’ve built.

Thanks so much for letting me know! It's always hard to tell when I put something out there if it just gets lost in the ether. I'm glad to hear it helped so many folks.

Re: Raft: Understandable Distributed Consensus (2014)

#53

Post by benbjohnson »

kfrzcode wrote: post #21
benbjohnson wrote: post #6

Author here. I'm happy to answer any questions although this project was from 10+ years ago so I could be a little rusty. Over the years I've been trying to find better ways to do this kind of visualization but for other CS topics. Moving to video is the most realistic option but using something like After Effects takes A LOT of time and energy for long-form visualizations. It also doesn't produce a readable output f…

What are your thoughts on Dr. Leemon Baird's Hedera Hashgraph? https://www.swirlds.com/downloads/SWIRLDS-TR-2016-01.pdf

I haven't read that paper but it seems like it's fixing a different problem of Byzantine fault tolerance. Most consensus systems that are internal for an organization don't have the Byzantine issue so it simplifies the problem.

Re: Raft: Understandable Distributed Consensus (2014)

#54

Post by alexnewman »

_russross wrote: post #19

I am in the minority who thinks Raft is overrated. I tried teaching Raft one year instead of Paxos but ended up switching back. While it was much easier to understand how to implement Raft, I think my students gained deeper insight when focusing on single-decision Paxos. There is a lightbulb moment when they first understand that consensus is a property of the system that happens first (and they can point at the mome…

You are telling me your students can safely implement single decree paxos.... I worked on a few paxos production implementations before the RAFT paper, single and multi decree. The idea that paxos, as it is to be implemented, is easy for students to understand... Well let me re-read the paper, but i assure you, raft was a big deal

Re: Raft: Understandable Distributed Consensus (2014)

#55

Post by prydt »

senderista wrote: post #42
prydt wrote: post #36

I've run a reading group for distributed systems for the last 2 years now and I do think that Raft is a better introduction to Consensus than any Paxos paper I have seen (I mean the Paxos Made Simple paper literally has bugs in it). But when I learned consensus in school, we used Paxos and Multi-Paxos and I do believe that there was a lot to be gained by learning both approaches. Heidi Howard has several amazing pape…

There are several Multi-Paxos papers (some of them dating before Raft) that are intended as guidance for implementers: https://paper-notes.zhjwpku.com/assets/pdfs/paxos_for_system... https://www.cs.cornell.edu/home/rvr/Paxos/paxos.pdf https://www.scs.stanford.edu/~dm/home/papers/paxos.pdf

Ah thank you. That is a good list although I personally dislike the "Paxos Made Moderately Complex" paper... I think it adds too many different roles for very little benefit. When implementing multi-Paxos for class, I used that paper and felt it was more trouble than it needed to be.

I'll check out the other two papers though! Also just looking around and I found this paper https://arxiv.org/pdf/1103.2408 [PDF] which looks useful as well.

Re: Raft: Understandable Distributed Consensus (2014)

#56

Post by jhanschoo »

Just earlier this month I was going through https://github.com/jepsen-io/maelstrom and following the demo implementing (a cut-corners version of) Raft, and I found it quite elucidating. The sample (ruby) code contains some bugs, and I had to use some understanding to fix them. (The bugs were of the kind where a dummy implementation from an earlier step isn't correctly changed)

Re: Raft: Understandable Distributed Consensus (2014)

#57

Post by hintymad »

_russross wrote: post #19

I am in the minority who thinks Raft is overrated. I tried teaching Raft one year instead of Paxos but ended up switching back. While it was much easier to understand how to implement Raft, I think my students gained deeper insight when focusing on single-decision Paxos. There is a lightbulb moment when they first understand that consensus is a property of the system that happens first (and they can point at the mome…

Is there any paper/handouts/video that explains Paxos in depth, especially its implementations and intuitions? Paxos Made Simple gave intuitive explanations, but I feel it still misses a lot of intricate details if I were to build Praxos for production use.

Re: Raft: Understandable Distributed Consensus (2014)

#59

Post by shepherdjerred »

prydt wrote: post #36

I've run a reading group for distributed systems for the last 2 years now and I do think that Raft is a better introduction to Consensus than any Paxos paper I have seen (I mean the Paxos Made Simple paper literally has bugs in it). But when I learned consensus in school, we used Paxos and Multi-Paxos and I do believe that there was a lot to be gained by learning both approaches. Heidi Howard has several amazing pape…

What’s your reading group?

I took a DS class and (poorly) implemented Paxos a few years ago. I’m curious about how others continue learning about DS.

Re: Raft: Understandable Distributed Consensus (2014)

#60

Post by MarkMarine »

throwawaymaths wrote: post #12
MarkMarine wrote: post #3

This is one of my favorite pieces of software engineering because it took something difficult and tried to design something easy to understand as a main criteria for success. The PHD Thesis has a lot more info about this if anyone is curious, it is approachable and easy to read: https://web.stanford.edu/~ouster/cgi-bin/papers/OngaroPhD.pd... I think this was core to Raft’s success, and I strive to create systems like…

Weirdly it's also kinda worse is better: raft is non-deterministic and has an unboundedly long election cycle time. IIRC: - it assumes no hysteresis in network latencies and if there is a hysteresis it's possible that elections can be deterministically infinite. - this fact and the use of raft in production has caused real, large scale network outages. Paxos is of course a beast and hard to understand. There is an al…

> this fact and the use of raft in production has caused real, large scale network outages.

Paxos as well, I remember full cloud GCP outage that had something to do with Paxos, and I can’t find the data on it but I thought there was a nasty bug in zookeeper paxos implementation.

That isn’t to say any of these are perfect or bug free, it’s made by humans and we’re going to make mistakes, but my experience implementing both was I had a working raft implementation and paxos baked my brain until I gave up.

I think everyone uses raft _because_ it was possible to implement for a working dev, so there are a number of implementations, and it’s easier to understand the phases the application is in.

I’ll check out VSR I appreciate the rec.

Post reply on HN