WPaxos: a wide area network Paxos protocol
muratbuffalo.blogspot.com
WPaxos: a wide area network Paxos protocol
1–10 of 20 posts
Re: WPaxos: a wide area network Paxos protocol
#2Need some non holiday time to read this fully, but it's worth noting that aside from Spanner, Cloud Datastore (using Megastore tech) and Cloud Firestore (using the same tech as Cloud Spanner) all do Paxos across several data centers over a wide area and achieve millions non batched writes/second.
Sometimes it can be hard to tease out the true trade-offs in the systems as they don't get as detailed acknowledgement, but from a quick read it looks like tail latency will will be worse.
Improvements in consensus algos often have issues, correct implementation is hard, but it can make previously impossible things tenable which I find exciting.
Re: WPaxos: a wide area network Paxos protocol
#3[deleted]
Re: WPaxos: a wide area network Paxos protocol
#4Need some non holiday time to read this fully, but it's worth noting that aside from Spanner, Cloud Datastore (using Megastore tech) and Cloud Firestore (using the same tech as Cloud Spanner) all do Paxos across several data centers over a wide area and achieve millions non batched writes/second. Sometimes it can be hard to tease out the true trade-offs in the systems as they don't get as detailed acknowledgement, bu…
The fact that Google is able to achieve good write performance with standard Paxos over wide areas is a direct result of their advanced network infrastructure and low latency data center interconnects. I imagine there are lots of people without a Google-class network who would benefit from WPaxos or something similar.
Re: WPaxos: a wide area network Paxos protocol
#5I think it's a huge positive development that there's a reference implementation as well, in go. At least helpful for evaluation, possibly for a production implementation also.
Re: WPaxos: a wide area network Paxos protocol
#6Need some non holiday time to read this fully, but it's worth noting that aside from Spanner, Cloud Datastore (using Megastore tech) and Cloud Firestore (using the same tech as Cloud Spanner) all do Paxos across several data centers over a wide area and achieve millions non batched writes/second. Sometimes it can be hard to tease out the true trade-offs in the systems as they don't get as detailed acknowledgement, bu…
The fact that Google is able to achieve good write performance with standard Paxos over wide areas is a direct result of their advanced network infrastructure and low latency data center interconnects. I imagine there are lots of people without a Google-class network who would benefit from WPaxos or something similar.
Our network definitely makes a big difference, but even we don't just use straight up vanilla Paxos.
Re: WPaxos: a wide area network Paxos protocol
#7What are example real world applications and/or use-cases where number of acceptors must be more than 3 or 5? Thanks.
Re: WPaxos: a wide area network Paxos protocol
#8It looks like the core of the idea is "object stealing". In most consensus algorithms you'll have a single leader, worldwide, that is in charge of sequencing all objects. Either that, or you partition objects across leaders and then coordinate across partitions.
In this algorithm you have multiple leaders, where a leader can be in charge of a particular object temporarily by "stealing" it. This brings control to the datacenter that needs it, reducing latency.
It's an interesting idea, and I haven't fully absorbed the paper. But I'll comment on it anyway. It's always easier to criticize when you're ignorant of the details :)
In general, I don't like these algorithms that only guarantee a partial ordering. In other words, instead of having a single global total ordering of all transactions, you have an ordering of operations that apply to a single object or key or a related group of keys. This means you can't know if object A got updated before object B. You can't know the total state of the system as of a snapshot in time, or as of the moment a node membership change happens. (Maybe. There might be workarounds for these issues, at the cost of some complexity.)
This paper is a good contribution, but we ain't there yet.
Re: WPaxos: a wide area network Paxos protocol
#9Need some non holiday time to read this fully, but it's worth noting that aside from Spanner, Cloud Datastore (using Megastore tech) and Cloud Firestore (using the same tech as Cloud Spanner) all do Paxos across several data centers over a wide area and achieve millions non batched writes/second. Sometimes it can be hard to tease out the true trade-offs in the systems as they don't get as detailed acknowledgement, bu…
The fact that Google is able to achieve good write performance with standard Paxos over wide areas is a direct result of their advanced network infrastructure and low latency data center interconnects. I imagine there are lots of people without a Google-class network who would benefit from WPaxos or something similar.
Does google (assuming Spanner) really use Paxos across WAN?
My understanding is, it does Two-Phase-Commit (across WAN) over fault-tolerant objects (located in a Paxos Group). Paxos Group could span across Data-Centers, but there it just uses normal Multi-Paxos -- perhaps with some extra leader election tricks.
Re: WPaxos: a wide area network Paxos protocol
#10It looks like the core of the idea is "object stealing". In most consensus algorithms you'll have a single leader, worldwide, that is in charge of sequencing all objects. Either that, or you partition objects across leaders and then coordinate across partitions. In this algorithm you have multiple leaders, where a leader can be in charge of a particular object temporarily by "stealing" it. This brings control to the…
Although I will say that this would handle a "follow the sun" workload pretty well.