Scaling Raft
cockroachlabs.com
Scaling Raft
1–10 of 13 posts
Re: Scaling Raft
#2Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others).
This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every node talking to every other node every few seconds.
What we really need is someone smart to come up with a consensus algorithm that doesn't need a heartbeat. Until then, it's two-phase commit if you want a reliable, large-scale (if not performant) system.
Right now, if you want a system that runs fast you use a consensus algorithm for shard metadata, but you do writes directly to the nodes without getting a consensus first. You run the risk of losing acknowledged writes, but it's the best you can do if you need speed.
Re: Scaling Raft
#3Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
(author of the original post here) Raft can be pretty relaxed about heartbeats. It doesn't, strictly speaking, require every node to talk to everyone else. All you really need is some health signal about the other nodes so you A) don't call an election while the leader is still alive and B) call an election promptly when a leader disappears. As a first step, we've reduced our heartbeat traffic from one per group to one per node, and we can reduce it further (e.g. by having each node send heartbeats only to a subset of its peers and sharing the results to other nodes).
Also, we are using both Raft and two-phase commit: Raft manages the consistency of individual ranges, but transactions that span multiple ranges require 2PC.
Re: Scaling Raft
#4Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
> The problem is the heartbeat; you can't have every node talking to every other node every few seconds.
The node does not have to talk with EVERY other nodes.
> What we really need is someone smart to come up with a consensus algorithm that doesn't need a heartbeat.
Well. Most consensus does not require heartbeat. Heartbeat is just the simplest way to detect a potential failure.
> if you want a system that runs fast
Consensus might affect the commit latency. It wont affect throughput that much though. But if you do not care, you can simply write to a consensus system without waiting for the commit. You can still get the serialization and replication.
Re: Scaling Raft
#5Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
Please clarify what you mean by "these consensus algorithms". Raft is merely an instance of a specific subset of C.A.s (published) to date.
Raft's design goals were primarily to be "understandable". A noble pedagogical goal in the abstract but misguided for use in production given the prevailing mindset [1].
> What we really need is someone smart ...
[1] Under 20 years of age only or will you accept advice from 50+ set? /G
Re: Scaling Raft
#6Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
Please clarify what you mean by "these consensus algorithms". Raft is merely an instance of a specific subset of C.A.s (published) to date. Raft's design goals were primarily to be "understandable". A noble pedagogical goal in the abstract but misguided for use in production given the prevailing mindset [1]. > What we really need is someone smart ... [1] Under 20 years of age only or will you accept advice from 50+ s…
I was thinking of Paxos and Raft, which are the two prevailing algorithms.
(I'm 53. Would be happy to hear your old-man theories :)
Re: Scaling Raft
#7Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
(author of the original post here) Raft can be pretty relaxed about heartbeats. It doesn't, strictly speaking, require every node to talk to everyone else. All you really need is some health signal about the other nodes so you A) don't call an election while the leader is still alive and B) call an election promptly when a leader disappears. As a first step, we've reduced our heartbeat traffic from one per group to o…
There's an interesting comment on the blog post:
> Does handling all range's consensus traffic in bulk would be the most effective way to handle this? Instead could all the ranges from a node be represented as a single state machine to be handled by plain etcd raft implementation, like the approach taken in spanner paper?
Do you have any comment on this? I've been trying to wrap my head around the implications. What's the downside?
Re: Scaling Raft
#8Earlier quoted context omitted.
(author of the original post here) Raft can be pretty relaxed about heartbeats. It doesn't, strictly speaking, require every node to talk to everyone else. All you really need is some health signal about the other nodes so you A) don't call an election while the leader is still alive and B) call an election promptly when a leader disappears. As a first step, we've reduced our heartbeat traffic from one per group to o…
There's an interesting comment on the blog post: > Does handling all range's consensus traffic in bulk would be the most effective way to handle this? Instead could all the ranges from a node be represented as a single state machine to be handled by plain etcd raft implementation, like the approach taken in spanner paper? Do you have any comment on this? I've been trying to wrap my head around the implications. What'…
I replied on the blog post. The downside is basically that if you have one giant consensus group, then to add a new replica you have to replicate that entire consensus group. This hurts load balancing and recovery times, since you can't scatter a dead node's ranges across all the other nodes in the cluster.
Re: Scaling Raft
#9Earlier quoted context omitted.
Please clarify what you mean by "these consensus algorithms". Raft is merely an instance of a specific subset of C.A.s (published) to date. Raft's design goals were primarily to be "understandable". A noble pedagogical goal in the abstract but misguided for use in production given the prevailing mindset [1]. > What we really need is someone smart ... [1] Under 20 years of age only or will you accept advice from 50+ s…
I was thinking of Paxos and Raft, which are the two prevailing algorithms. (I'm 53. Would be happy to hear your old-man theories :)
You are thinking about consistent RSM. Paxos and Raft are not the similar thing and not even at the same level.
Epaxos, along with other master-less RSM, does not require heartbeat. It is not about smart people, it is about how you want to implement your system based on the requirement.
Re: Scaling Raft
#10Yup, this is the key problem with these consensus algorithms. They don't handle shards very well. Most systems these days either punt and don't do updates via the consensus algorithm (Apache Kafka), or they use two-phase commit (FoundationDB, many others). This is important work, but I don't think they're really going to succeed is making a truly large-scale system. The problem is the heartbeat; you can't have every…
I'm not sure how you could call it a consensus algorithm without putting updates through the state machine, since that's essentially all Raft does.
"The problem is the heartbeat; you can't have every node talking to every other node every few seconds"
So nodes are just to assume that other nodes are alive/reachable/functioning? How would you propose detecting node health without some form of heartbeat?
"... but you do writes directly to the nodes without getting a consensus first. You run the risk of losing acknowledged writes..."
This is the equivalent of the default MongoDB write strategy. You probably don't want it. In the majority of use cases you really, really want writes to be committed to the replicated state machine on a quorum of nodes before they are acknowledged back to the client.