Live data from Hacker News

Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

cockroachlabs.com

11–20 of 41 posts

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#11
post #6

I like the elegance and simplicity of two-phase commits. I didn't understand the criticism in the article; maybe it's something specific to CockroachDB. In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts. In one of my open source projects with RethinkDB (which doesn't natively supp…

This is about reducing the number of message delays before the commit succeeds. Failure scenarios have to be handled to be correct, but this is a performance optimization primarily from what I can see.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#12
post #6

I like the elegance and simplicity of two-phase commits. I didn't understand the criticism in the article; maybe it's something specific to CockroachDB. In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts. In one of my open source projects with RethinkDB (which doesn't natively supp…

> Whenever a server failed and restarted, it would pick up processing from the last successful commit. If a server did not restart, the worker count would be updated and remaining workers would redistribute the partitions among themselves based on the shard keys of the records.

You don't describe how you come to a global view of which is the "last successful commit". Do you mean that the co-ordinator recovers (such as from its own log) and uses that information, or do you mean you use local information? In the latter case, what you describe doesn't sound like it preserves transaction atomicity across node crashes.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#13
post #5
post #3

Earlier quoted context omitted.

Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment. 1. A controller starts a transaction. 2. Some code looks up a record in a database. 3. Based on the results of that lookup we run 1 of two possible writes to a table in that database. Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which its…

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

Nobody in industry understood how to apply Calvin until we did at Fauna. That is the only reason; the rest is engineering path dependence.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#14
post #12
post #6

I like the elegance and simplicity of two-phase commits. I didn't understand the criticism in the article; maybe it's something specific to CockroachDB. In my experience with two-phase commits, if the system crashes before a transaction is fully processed and committed, it should be fully reprocessed (from scratch) once the server restarts. In one of my open source projects with RethinkDB (which doesn't natively supp…

> Whenever a server failed and restarted, it would pick up processing from the last successful commit. If a server did not restart, the worker count would be updated and remaining workers would redistribute the partitions among themselves based on the shard keys of the records. You don't describe how you come to a global view of which is the "last successful commit". Do you mean that the co-ordinator recovers (such a…

In RethinkDB, a table can be split into shards and each shard has at most one master and multiple replicas. With one master to perform all writes for a shard, there is no need for a co-ordinator. If the master of a shard goes down, a replica can be promoted as the master. For writes, RethinkDB prioritizes consistency over availability so a server failure may carry some downtime in terms of writes but reads can be configured to be highly available (though they could potentially be out of date).

But there is definitely a downside that if a RethinkDB master server fails immediately after a write (before it propagated to a replica) and does not recover, then there can be some data loss (but only the most recent writes on that shard).

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#15
post #12

Earlier quoted context omitted.

> Whenever a server failed and restarted, it would pick up processing from the last successful commit. If a server did not restart, the worker count would be updated and remaining workers would redistribute the partitions among themselves based on the shard keys of the records. You don't describe how you come to a global view of which is the "last successful commit". Do you mean that the co-ordinator recovers (such a…

In RethinkDB, a table can be split into shards and each shard has at most one master and multiple replicas. With one master to perform all writes for a shard, there is no need for a co-ordinator. If the master of a shard goes down, a replica can be promoted as the master. For writes, RethinkDB prioritizes consistency over availability so a server failure may carry some downtime in terms of writes but reads can be con…

So if I'm reading this right, the master process can lose writes (if there's permanent failure). So taking your 2PC scheme above with the 'settled' flag, you can respond to the client that the txn has been committed once you've written the flag, but this commit marker can also just be entirely lost? (again, permanent failure)

If this 'settled' flag exists on each 'shard' instead of just the coordinator, any random subset of those too can be lost? I don't understand what's going on here, or what guarantees this 2PC implementation provides.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#16
post #5

Earlier quoted context omitted.

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

Nobody in industry understood how to apply Calvin until we did at Fauna. That is the only reason; the rest is engineering path dependence.

if you don’t mind sharing, I’m curious to learn more what was the main challenge was applying Calvin.

Thanks!

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#17
post #5
post #3

Earlier quoted context omitted.

Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment. 1. A controller starts a transaction. 2. Some code looks up a record in a database. 3. Based on the results of that lookup we run 1 of two possible writes to a table in that database. Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which its…

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

> That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result.

If that function can still do everything the client did and the client still has to wait for a transaction - you are only eliminating interactive communications overhead, not actually improving or simplifying anything fundamentally. There is still consensus, coordination happening during that waiting and all of this is still fundamentally incompatible with computers spread across the globe communicating over unreliable links.

What would actually be a big improvement is eliminating waiting for coordination, but would require some change in programming model [1].

> The Calvin way is so much easier, it seems like there must be some very good reason that it's not what CockroachDB does.

It's just not "so much easier", that's the reason.

[1] https://arxiv.org/pdf/1901.01930.pdf

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#18
post #5
post #3

Earlier quoted context omitted.

Interactive may be a little misleading here. Consider a common case in any java application at my current place of employment. 1. A controller starts a transaction. 2. Some code looks up a record in a database. 3. Based on the results of that lookup we run 1 of two possible writes to a table in that database. Knowing what kind of write you will do in the transaction requires knowing the result of the lookup which its…

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked.

In talking with others, my best guess as to why we've seen relatively few implementations of it in the wild is that it is just less well understood compared to 2PC, so misconceptions propagate. The original paper focuses on how it works, rather than how to apply it in detail to generic transaction processing, which perhaps is a shame in hindsight considering that is where most of the confusion lies, IMHO.

For example, there is no reason stemming from Calvin that FaunaDB cannot provide full SQL-style session transactions. We chose not to implement them because they aren't a good fit for the core use-cases the system currently targets. Specifically, interactive transactions are too chatty for client-server interactions over the internet where link latency dominates an application's perceived speed: Instead, FaunaDB's interface encourages packing as much logic into as few requests as possible. (But I suppose that's a topic for another comment thread.)

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#19

Earlier quoted context omitted.

Nobody in industry understood how to apply Calvin until we did at Fauna. That is the only reason; the rest is engineering path dependence.

if you don’t mind sharing, I’m curious to learn more what was the main challenge was applying Calvin. Thanks!

See @freels’ reply above.

I think it is fair to say that the Calvin paper is visibly incomplete and expresses some constraints in a way that makes them seem insurmountable when they are not; specifically, they are only constraints within the log, but do not constrain the database experience overall.

Applying Calvin to traditional RDBMS workloads was a very unlikely creative exercise because it required questioning these explicit constraints.

The Spanner paper also leaves a lot unexplained, but it is less obvious until you are too far down the path to turn back. After all, it worked for Google. Calvin did not have that real-world proof. Combine that with the pessimism of the paper itself and nobody was willing to pick it up.

Re: Parallel Commits: A New Atomic Commit Protocol for Distributed Transactions

#20
post #18
post #5

Earlier quoted context omitted.

Exactly. What Calvin (and FaunaDB, I think) does is allow you to run an arbitrary function which takes input on the database. That function can do whatever it wants, including any number of reads, writes, and arbitrary logic based on the reads. But critically, it can't talk back to the calling client, except for to send the final result. This allows you to implement the pattern you describe, which I agree is common,…

Calvin has been an elegant protocol to work with in practice, and has pretty radically simplified FaunaDB's implementation of transactions compared to classic 2PC. Writes are committed in one global communication exchange, read isolation is pretty straightforward, and not requiring transaction recovery cuts out a significant amount of complexity which tends to be overlooked. In talking with others, my best guess as t…

[deleted]
Post reply on HN