Live data from Hacker News

Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

fauna.com

61–70 of 105 posts

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#61
post #59

The paper contains this claim: > FaunaDB is an elegant, software-only solution for achieving global ACID transactions, with complete guarantees of serializability and consistency. The paper makes it sound like FaunaDB claims strict serializability for all transactions (like Spanner). This means that if Txn(A) ends before Txn(B) begins, then Txn(B) must be guaranteed to see all data written by Txn(A). However, what if…

Read-only transactions are serializable by default but can be upgraded to strict serializability, either from the perspective of a single client by passing a transaction timestamp that serves as a causal token, which is free and happens by default with our drivers, or globally, by opting in to the log at the price of additional latency.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#62

This is a weakness imo: `client.query( q.get( q.match( q.index("posts_by_title"), "My cat and other marvels" ) ))` That you have to specify the index in a query is a regression imo at least I've been spoiled by not having to have to do so in when using Mongo or SQL databases since the query engine will more often than not find the right index.

This a design choice to guarantee predictable performance at scale, not an architectural limitation, and subject to revision. You don't want your service to collapse when the query optimizer suddenly doesn't choose the right index anymore. That was a hard lesson learned for us at Twitter.

Is not possible to make the query optimizer deterministic?

I think is better that every time I see a query it do exactly the same steps. Only when testing/monitoring say so, I hint it.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#63
post #58

Earlier quoted context omitted.

The log segments committed by FaunaDB contain batches of transactions, which means our throughput is constrained not by our consensus protocol, but rather by conflicts as transactions are resolved. The benchmarked 3300 transactions/second mentioned is for complex transactions with dozens of reads and writes. Additionally, read-only transactions are not run through the transaction pipeline, since they can be served co…

Can you partition your batch transactions, so that all up to the conflict succeed?

My understanding is yes, we commit to the log in batches, but abort conflicts at the transaction level. So only the conflicts have to suffer the retry loop, everything else is durably committed.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#64
post #19

We're here, ready for your questions to be consistently replicated across the WAN at the lowest latency information science allows ;-)

Not a database person, so I might be missing context, but why is FaunaDB the only implementation of this? Intuitively, it seems to be exactly the way I would do a distributed database - distributed transactional storage with a global ordering of transactions that are based on ids rather than timestamps. The only thing that I wouldn't intuitively come up with is the actual consensus algorithm, but that already existed…

This is such a niche subcategory of distributed databases [1], that it's a miracle even a few of them exist and you want more implementations. Intuitively distributed systems try to avoid consensus/coordination, as doing those things over public internet is not that useful.

[1] https://trends.google.com/trends/explore?date=2016-09-20%202... compare that to some other distributed database https://trends.google.com/trends/explore?date=2016-09-20%202...

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#66
post #19

We're here, ready for your questions to be consistently replicated across the WAN at the lowest latency information science allows ;-)

Not a database person, so I might be missing context, but why is FaunaDB the only implementation of this? Intuitively, it seems to be exactly the way I would do a distributed database - distributed transactional storage with a global ordering of transactions that are based on ids rather than timestamps. The only thing that I wouldn't intuitively come up with is the actual consensus algorithm, but that already existed…

There's a few other research projects based on the same concept of a distributed log and an explicit sequencer such as Corfu: https://github.com/CorfuDB/CorfuDB

As far as why it's not more prevalent, all of this research is work from the last decade or less, and it takes a long time to develop a database ready for production use.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#67
post #64
post #19

Earlier quoted context omitted.

Not a database person, so I might be missing context, but why is FaunaDB the only implementation of this? Intuitively, it seems to be exactly the way I would do a distributed database - distributed transactional storage with a global ordering of transactions that are based on ids rather than timestamps. The only thing that I wouldn't intuitively come up with is the actual consensus algorithm, but that already existed…

This is such a niche subcategory of distributed databases [1], that it's a miracle even a few of them exist and you want more implementations. Intuitively distributed systems try to avoid consensus/coordination, as doing those things over public internet is not that useful. [1] https://trends.google.com/trends/explore?date=2016-09-20%202... compare that to some other distributed database https://trends.google.com/tre…

The property that attracts the most mission critical use cases is the ability to keep transactions rolling without loss of guarantees, even while hitting the typical cloud failure modes. Some applications can't make tradeoffs between correctness and global availability, for them FaunaDB's mainframe-like capabilities are key.

For the average application, the benefit is that you don't need to write code to address database drift / eventual consistency. I expect Fauna-like systems to gain popularity as more developers come to expect cloud-native solutions to offer global ACID transactions.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#68
post #59

The paper contains this claim: > FaunaDB is an elegant, software-only solution for achieving global ACID transactions, with complete guarantees of serializability and consistency. The paper makes it sound like FaunaDB claims strict serializability for all transactions (like Spanner). This means that if Txn(A) ends before Txn(B) begins, then Txn(B) must be guaranteed to see all data written by Txn(A). However, what if…

Read-only transactions are serializable by default but can be upgraded to strict serializability, either from the perspective of a single client by passing a transaction timestamp that serves as a causal token, which is free and happens by default with our drivers, or globally, by opting in to the log at the price of additional latency.

But opting into the log for read-only txns would destroy performance for many apps, both in terms of latency and throughput. This is rarely practical since so many apps are read-heavy. I assume you don't recommend that to your customers, especially in global deployments.

Causal tokens are practical from a performance point of view, but do require application awareness and changes, which presents a different set of issues. Most application developers would not appreciate the subtleties and just use the default behavior, which does not guarantee "read your own writes". And causal tokens don't prevent the more general case of stale reads.

This would mean that in the default FaunaDB configuration, applications could see anomalies:

1. Inability to read own writes - described above, with the additional clarification that it can't be fully solved at the driver level. For example, say I call a web API that creates a blog post and returns a PostID. I then call another API that edits that post, passing it the PostID I got back from the first. If FaunaDB is in use, the second call could fail to find the PostID, because I happened to hit a different replica that did not yet have it.

2. Stale reads - could be arbitrary levels of staleness if replica/coordinator haven't been able to sync to the global log. If a coordinator node and a replica were stuck in their own little partition, cut off from the rest of the cluster, they'd still happily serve stale reads (until perhaps they detect a partition and shut down?).

These read anomalies aren't usually considered compatible with a consistent system, but instead are hallmarks of an eventually consistent system. It seems misleading to make an unqualified 100% consistency claim when it doesn't apply to read-only txns by default, and when the fully consistent mode is not practical to use in global deployments.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#69
post #62

Earlier quoted context omitted.

This a design choice to guarantee predictable performance at scale, not an architectural limitation, and subject to revision. You don't want your service to collapse when the query optimizer suddenly doesn't choose the right index anymore. That was a hard lesson learned for us at Twitter.

Is not possible to make the query optimizer deterministic? I think is better that every time I see a query it do exactly the same steps. Only when testing/monitoring say so, I hint it.

Perhaps. What if you add a new index and it changes the optimization plan of existing queries that you didn’t anticipate? There are a lot of edge cases, but we are not religiously opposed to making the optimizer smarter.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#70
post #10
post #8

This looks like a fascinating approach. Unfortunately I’m not well versed enough to intelligently compare it to alternatives. Any plans to have Jepsen / Aphyr conduct a rigorous test and write a report on the results?

We have a whole suite of internal correctness verification tools, in addition to which we are also using Jepsen. We don't have an official report out yet, but you can get a preview of our progress here: https://fauna.com/blog/verifying-transactional-consistency-w...

Might want to look into methods FoundationDB was using:

https://abdullin.com/foundationdb-is-back/

Might be useful to your team. One of rare times I was surprised and impressed by a new company's verification process. I look forward to reading your tesm's write-up of techniques and lessons learned, too.

Post reply on HN