Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

cockroachlabs.com

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#21

Post by [deleted] »

zzzcpan wrote: post #17
aboodman wrote: post #5

Earlier quoted context omitted.

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

> That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. If that function can still do everything the client did and the client still has to wait for a transaction - you are only eliminating interactive communications overhead, not actually improving or simplif…

[deleted]

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#22

Post by dbuser99 »

freels wrote: post #18
aboodman wrote: post #5

Earlier quoted context omitted.

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked. In talking with others, my best guess as t…

Would that include transactions for which the reads can query the whole database as opposed to a predetermined set of rows?

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#23

Post by Pxtl »

aboodman wrote: post #8
foota wrote: post #7

Earlier quoted context omitted.

Wouldn't you have to implement a DSL and all the parts related to it for that to work? Also things like serialization cost. What if I have a large local in memory structure I want to base my query results off of? Funny enough this is kind of similar to the problem solved by things like apache beam.

No you'd need a complete programming language. But SQL is basically one already, most variants are already turing-complete. You'd also have to provide the programming environment with concepts of cursors and so on so they could page through data efficiently.

Every procedural layer I've ever used bolted onto SQL (pl/SQL, t-sql) has been absolute goddamned agony to use.

SQL is a good (if dated) language for relational access and manipulation, but awful for procedural scripting.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#25

Post by ComodoHacker »

From what I understand, with this new commit protocol they managed to improve response time of a writing transaction by shifting some work of determining its final status to the readers. Am I understanding correctly that readers' performance will degrade by the same amount?

While this is an achievement which some scenarios will definitely benefit from, like bulk loading or updating secondary indexes as mentioned in the article, what about other scenarios where read performance is more important, like the ones where there are much more readers than writers? Shouldn't there be a configuration option which commit protocol to use?

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#26

Post by irfansharif »

ComodoHacker wrote: post #25

From what I understand, with this new commit protocol they managed to improve response time of a writing transaction by shifting some work of determining its final status to the readers. Am I understanding correctly that readers' performance will degrade by the same amount? While this is an achievement which some scenarios will definitely benefit from, like bulk loading or updating secondary indexes as mentioned in t…

> Am I understanding correctly that readers' performance will degrade by the same amount?

Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself.

In normal conditions however, once the coordinator node receives acknowledgement for the successful persisting of all write intents and the "txn STAGING" record, it can simply record "txn COMMITTED" in memory (and return to the client, send off async intent resolution procedures, etc.) Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. This is all safe because the commit marker is not simply stored on the coordinator node, it's a distributed condition and can be reconstructed by any observer even if the coordinator failed.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#27

Post by ComodoHacker »

irfansharif wrote: post #26
ComodoHacker wrote: post #25

From what I understand, with this new commit protocol they managed to improve response time of a writing transaction by shifting some work of determining its final status to the readers. Am I understanding correctly that readers' performance will degrade by the same amount? While this is an achievement which some scenarios will definitely benefit from, like bulk loading or updating secondary indexes as mentioned in t…

> Am I understanding correctly that readers' performance will degrade by the same amount? Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself. In normal conditions however, once the coordinator node receives acknowledgement for…

>Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory.

Another roundtrip performed by reader rather than writer? That's what I'm talking about.

Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#28

Post by cryptica »

irfansharif wrote: post #15
cryptica wrote: post #14

Earlier quoted context omitted.

In RethinkDB, a table can be split into shards and each shard has at most one master and multiple replicas. With one master to perform all writes for a shard, there is no need for a co-ordinator. If the master of a shard goes down, a replica can be promoted as the master. For writes, RethinkDB prioritizes consistency over availability so a server failure may carry some downtime in terms of writes but reads can be con…

So if I'm reading this right, the master process can lose writes (if there's permanent failure). So taking your 2PC scheme above with the 'settled' flag, you can respond to the client that the txn has been committed once you've written the flag, but this commit marker can also just be entirely lost? (again, permanent failure) If this 'settled' flag exists on each 'shard' instead of just the coordinator, any random su…

Actually, I may be mistaken about my previous commment. I'm not completely sure if this loss of recent data would happen as I've described. It depends on client implementation. For example, a client could wait for a write to propagate to at least 1 replica before telling the caller that the data was inserted successfully. This is an implementation detail I'm not sure about.

Also the settled flag exists on each record, not each shard. A shard is typically made up of multiple unsettled records. Each worker is assigned to a shard using a hash function so it's deterministic and the worker only processes unsettled transactions from their own shard.

Also I said something else misleading in one of my previous comments. In my case, the shard key of each record (which determines which shard a record belongs to) was not based on its own record ID but on the account ID of the user who owns that record. So effectively the sharding was happening based on user accounts and it was designed so that the records created by an account could be processed independently of records created by a different account.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#29

Post by jlokier »

ComodoHacker wrote: post #27
irfansharif wrote: post #26

Earlier quoted context omitted.

> Am I understanding correctly that readers' performance will degrade by the same amount? Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself. In normal conditions however, once the coordinator node receives acknowledgement for…

>Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. Another roundtrip performed by reader rather than writer? That's what I'm talking about. Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure.

So:

- Writes are faster due to fewer round trips in normal running.

- Reads are the same speed in normal running.

- After coordinator failure events, the new coordinator starts to clean up left-over intents asynchronously (nothing specific is waiting for it to finish).

- Reads are slowed by extra round trips in the short time after a failure event, until the left-over intents are cleaned up. But only in that time period, and only for those ranges touched by transactions during the failure event.

- The cost of extra round trips done by the reader just during recovery is much less important than the round trips that happened on every cross-region write with the older algorithm.

- But if you care about read latency being consistent all the time, including during recovery from coordinator failure events, maybe you need a more sophisticated high-availability configuration for the logical coordinator.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#30

Post by jlokier »

cryptica wrote: post #6

I like the elegance and simplicity of two-phase commits. I didn't understand the criticism in the article; maybe it's something specific to CockroachDB. In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts. In one of my open source projects with RethinkDB (which doesn't natively supp…

> In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts.

Then the problem becomes, in a fully distributed system how do you know which was the last successful commit before the system crashed?

That 'settled' flag you mention may be set in the coordinator node's storage just before the coordinator crashed, but not communicated to any other node because of the crash.

So the other nodes have to wait for the coordinator to restart, and replay its storage, to find out if that transaction was successful.

That pause can be quite long, especially if it involves a coordinator rebooting, and really long if it involves resynchronising a RAID, checking filesystems/DB integrity, etc.

The other nodes could decide it doesn't matter, exclude the coordinator from the cluster, and reprocess starting from an older transaction. But then they have to agree which older transaction - a distributed consensus problem.

This is certainly possible, but I wouldn't call it simple, and it wouldn't be two-phase commit any more.

Restarting older transactions tends to be visible to clients as well, because they may see transaction retries, and it's often desirable to minimise the number of those from a distributed database for minor failures (such as one node harmlessly going offline). Among other things, they can cause load spikes in the large system, and may exercise retry corner cases that don't come up normally, surfacing bugs that shouldn't be there in application code, but are.

Post reply on HN