Scalable but Wasteful, or why fast replication protocols are slow
1–10 of 26 posts
Re: Scalable but Wasteful, or why fast replication protocols are slow
#2Re: Scalable but Wasteful, or why fast replication protocols are slow
#3So, why?
Re: Scalable but Wasteful, or why fast replication protocols are slow
#4So, why?
Re: Scalable but Wasteful, or why fast replication protocols are slow
#5If your raft state machines are doing IO via some write through cache (which they often are) then having specific machines do specific jobs can increase the cache quality. I.e. your leader node can have a better cache for your write workload, whilst your follower nodes can have better caches for your read workload.
This may lead to higher throughput (yay) but then also leave you vulnerable to significant slow-downs after leader elections (boo).
What makes sense will depend on your use case, but I personally agree with the author that multiple simple raft/paxos groups scheduled across nodes by some workload aware component might be the best of both worlds.
Re: Scalable but Wasteful, or why fast replication protocols are slow
#6So, why?
A leaderless protocol is more complex and requires greater IO and CPU resources per unit of work. This is the hidden cost. The question is if the cost is reasonably offset by higher performance (throughput) offered by leaderless consensus. OP is arguing that the benefit is not deemed worth the cost outside of academia and this is why e.g EPaxos is not adopted.
Re: Scalable but Wasteful, or why fast replication protocols are slow
#7Does leader = master here? My first reaction is that this is a multi-master system but I can't quite unpack "opportunistic coordinator".
Re: Scalable but Wasteful, or why fast replication protocols are slow
#8Re: Scalable but Wasteful, or why fast replication protocols are slow
#9The hypothesis in the article could be correct (that industry is not adopting new academic innovations because they fail in the real world). Based on my experience in this industry, though, it could just be that there isn't a super strong connection between academia and the people implementing these kinds of systems. I've had many conversations with my academically-minded friends where they're astonished that we have…