Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

cockroachlabs.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#31

Post by evanweaver »

dbuser99 wrote: post #22
freels wrote: post #18

Earlier quoted context omitted.

Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked. In talking with others, my best guess as t…

Would that include transactions for which the reads can query the whole database as opposed to a predetermined set of rows?

The SQL support? Yes, even that.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#32

Post by irfansharif »

ComodoHacker wrote: post #27
irfansharif wrote: post #26

Earlier quoted context omitted.

> Am I understanding correctly that readers' performance will degrade by the same amount? Not quite. The "slow path" talked about is only applicable when the coordinator node is unavailable (presumably a rare event). If it's unavailable, there's nobody left to clean up the STAGING txn record, so the reader is tasked to do it itself. In normal conditions however, once the coordinator node receives acknowledgement for…

>Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. Another roundtrip performed by reader rather than writer? That's what I'm talking about. Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

> Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

Actually, welp, I believe this is closer to the truth in implementation. But consider how read performance compares before and after introduction of Parallel Commits. Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution (on the coordinator node of the other txn, or on the node the intent was seen). They simply continue doing the same in Parallel Commits, there's no extra latency added to the read path (except when there is failure, but even then as soon as the earlier txn is recovered, future readers no longer get stuck).

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#33

Post by irfansharif »

jlokier wrote: post #29
ComodoHacker wrote: post #27

Earlier quoted context omitted.

>Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. Another roundtrip performed by reader rather than writer? That's what I'm talking about. Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure. So: - Writes are faster due to fewer round trips in normal running. - Reads are the same speed in normal running. - After coordinator failure events, the new coordinator starts to clean up left-over intents asynchronously (nothing specific is waiting for it to finish). - Reads…

> If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure.

Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how.

> After coordinator failure events, the new coordinator starts to clean up left-over intents asynchronously (nothing specific is waiting for it to finish).

The cleanup happens by readers on demand, there's no separate global coordinator scanning the keyspace and resolving old write intents.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#34

Post by jlokier »

irfansharif wrote: post #33
jlokier wrote: post #29

Earlier quoted context omitted.

If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure. So: - Writes are faster due to fewer round trips in normal running. - Reads are the same speed in normal running. - After coordinator failure events, the new coordinator starts to clean up left-over intents asynchronously (nothing specific is waiting for it to finish). - Reads…

> If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure. Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how. > After coordinator failure events, the new coordinator starts to clean up left-ove…

> Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how.

Thanks, that was a helpful comment.

> The cleanup happens by readers on demand, there's no separate global coordinator scanning the keyspace and resolving old write intents.

Oh, that's a little surprising. I assumed the coordinator(s) did so because asynchronous cleanup is mentioned numerous times in the article, but upon closer scrutiny I see now that it only applies in the after phase of transactions without a failure.

Would that scanning, analogous to RAID "resilvering" subject to write-intent ranges to limit the keyspace regions scanned, usefully improve read latencies later?

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#35

Post by ComodoHacker »

irfansharif wrote: post #32
ComodoHacker wrote: post #27

Earlier quoted context omitted.

>Any subsequent read requests that observes left over intents (yet to be resolved) are pointed to the coordinator node, which can simply consult the "txn COMMITTED" record in memory. Another roundtrip performed by reader rather than writer? That's what I'm talking about. Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node.

> Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node. Actually, welp, I believe this is closer to the truth in implementation. But consider how read performance compares before and after introduction of Parallel Commits. Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution (on the coo…

>Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution

Again, I understand it differently, but maybe I'm wrong. A reader upon encountering unresolved intent looks up corresponding transaction record. Before Parallel Commits, it's either marked COMMITTED or PENDING. If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's no waiting here.

Now with parallel commits, transaction record can also be marked STAGING, in which case the reader cannot determine if it's commited without additional work and/or waiting (the author doesn't go much into details).

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#36

Post by nvanbenschoten »

ComodoHacker wrote: post #35
irfansharif wrote: post #32

Earlier quoted context omitted.

> Though I understand it differently, as reader just waits until all writes and "txn COMMITTED" record arrive at it's node. Actually, welp, I believe this is closer to the truth in implementation. But consider how read performance compares before and after introduction of Parallel Commits. Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution (on the coo…

>Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution Again, I understand it differently, but maybe I'm wrong. A reader upon encountering unresolved intent looks up corresponding transaction record. Before Parallel Commits, it's either marked COMMITTED or PENDING. If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's n…

> If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's no waiting here.

I think this is where the confusion is coming from. You're correct that a read can simply ignore writes, even pending ones, at higher timestamps due to MVCC. This improves transaction concurrency.

However, if a read finds a provisional write (an intent) at a lower timestamp, it can't just ignore it. It needs to know whether to observe the write or not. So it looks up the write's transaction record and may have to wait. If the write transaction is not finalized then it needs to either wait on the transaction to finish or force the transaction's timestamp up above its read timestamp. This is true regardless of parallel commits or not.

What parallel commits gets us is a faster path to transaction commit, as irfansharif pointed out below. So the write can not only be committed faster with parallel commits, but it can also be resolved faster to get out of other reads' ways. In that way, it improves both the synchronous latency profile and the contention footprint of transactions, assuming no coordinator failures.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#37

Post by irfansharif »

jlokier wrote: post #34
irfansharif wrote: post #33

Earlier quoted context omitted.

> If I understood correctly, the extra round trip on the reader only occurs with left-over intents, which are the product of an earlier failure. Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how. > After coordinator failure events, the new coordinator starts to clean up left-ove…

> Left over intents are also visible for an ongoing txn, before those intents are resolved. But there's no added latency in the read path, I've commented elsewhere in the thread to explain how. Thanks, that was a helpful comment. > The cleanup happens by readers on demand, there's no separate global coordinator scanning the keyspace and resolving old write intents. Oh, that's a little surprising. I assumed the coordi…

> I see now that it only applies in the after phase of transactions without a failure.

Yep.

> Would that scanning, analogous to RAID "resilvering" subject to write-intent ranges to limit the keyspace regions scanned, usefully improve read latencies later?

I think it's just a better design to have it done on demand. The keyspace is large and failures are rare, and when one of these zombie intents are happened upon, the very first reader addressing it resolves it for all subsequent readers. A global scan would improve read latencies later, but not by much and not for many readers.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#39

Post by irfansharif »

grogers wrote: post #38

Given how critical preventing future intent writes is to the protocol to ensure safety during recovery, it'd be nice to have more detail on how that works. Calling it an in memory data structure doesn't exactly inspire confidence.

Agreed, we should be talking about the "timestamp cache" in more detail generally. While I'm here, looking at [1] helped me confirm how everything is kosher despite being in-memory. The timestamp cache pessimistically maintains a "low water mark", this always ratchets up monotonically and represents the earliest access timestamp of any key in the range. Writes happening at timestamps lower than this water mark are not let through, and bumping this watermark past the point of the observed write intent's timestamp is how slow inflight write intents are aborted by recovering read requests. On server restart, the timestamp cache is initialized with a low water mark of the current system time + maximum clock offset, so "future" write intents (or more accurately: write intents sent in the past but previously stuck in transit) are simply rejected.

[1]: https://github.com/cockroachdb/cockroach/blob/master/pkg/sto...

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#40

Post by ComodoHacker »

nvanbenschoten wrote: post #36
ComodoHacker wrote: post #35

Earlier quoted context omitted.

>Before, when readers happen over extant 2PC "prepare" phase markers, they would still have to wait for txn resolution Again, I understand it differently, but maybe I'm wrong. A reader upon encountering unresolved intent looks up corresponding transaction record. Before Parallel Commits, it's either marked COMMITTED or PENDING. If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's n…

> If it's PENDING, reader just ignores it and skips its data, since they use MVCC. There's no waiting here. I think this is where the confusion is coming from. You're correct that a read can simply ignore writes, even pending ones, at higher timestamps due to MVCC. This improves transaction concurrency. However, if a read finds a provisional write (an intent) at a lower timestamp, it can't just ignore it. It needs to…

Thank you for clarification.
Post reply on HN