Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

fauna.com

Post reply on HN
Print view View original

51–60 of 105 posts

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#51

Post by wallflower »

gigatexal wrote: post #46

This is a weakness imo: `client.query( q.get( q.match( q.index("posts_by_title"), "My cat and other marvels" ) ))` That you have to specify the index in a query is a regression imo at least I've been spoiled by not having to have to do so in when using Mongo or SQL databases since the query engine will more often than not find the right index.

Mongo has issues with its query optimizer. I’ve seen this happen before, killing performance. The fix was to give the query a hint on the correct index to use.

A similar issue reported

https://jira.mongodb.org/plugins/servlet/mobile#issue/SERVER...

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#52

Post by gigatexal »

wallflower wrote: post #51
gigatexal wrote: post #46

This is a weakness imo: `client.query( q.get( q.match( q.index("posts_by_title"), "My cat and other marvels" ) ))` That you have to specify the index in a query is a regression imo at least I've been spoiled by not having to have to do so in when using Mongo or SQL databases since the query engine will more often than not find the right index.

Mongo has issues with its query optimizer. I’ve seen this happen before, killing performance. The fix was to give the query a hint on the correct index to use. A similar issue reported https://jira.mongodb.org/plugins/servlet/mobile#issue/SERVER...

So I can see the justification for adding a bit of work to the developer or query writer to pick the right index for the document would mean consistent query performance every time (since it's a document anyway finding the right index should be pretty easy I would imagine). I can see why one would choose this then.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#53

Post by evanweaver »

gigatexal wrote: post #47
evanweaver wrote: post #45

Earlier quoted context omitted.

Not quite what you are asking, but some customers with case studies on the site: https://fauna.com/customers

Unbiased in that the article we are all referencing is coming from you guys, the creators, you can't help but show it in a positive light and I don't fault you for it. Just I've become so jaded of late that I am looking for people who have used it in the wild. I also signed up for the cloud account -- your pricing terms are some of the best I've ever seen, kudos -- so that I can check out the API some more.

Thanks; what do you like about the pricing?

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#54

Post by [deleted] »

gigatexal wrote: post #47
evanweaver wrote: post #45

Earlier quoted context omitted.

Not quite what you are asking, but some customers with case studies on the site: https://fauna.com/customers

Unbiased in that the article we are all referencing is coming from you guys, the creators, you can't help but show it in a positive light and I don't fault you for it. Just I've become so jaded of late that I am looking for people who have used it in the wild. I also signed up for the cloud account -- your pricing terms are some of the best I've ever seen, kudos -- so that I can check out the API some more.

[deleted]

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#55

Post by gigatexal »

evanweaver wrote: post #53
gigatexal wrote: post #47

Earlier quoted context omitted.

Unbiased in that the article we are all referencing is coming from you guys, the creators, you can't help but show it in a positive light and I don't fault you for it. Just I've become so jaded of late that I am looking for people who have used it in the wild. I also signed up for the cloud account -- your pricing terms are some of the best I've ever seen, kudos -- so that I can check out the API some more.

Thanks; what do you like about the pricing?

The free forever on premise pricing for personal use to the 90-trial in production even in commercial use. And I thought the free threshold for the cloud reasonable as I can forgo all the setup work and just use the cloud isntance for free if my usage falls under the right level. Anything that saves me time but then doesn’t cost me anything while learning it / trying it out is a win in my book.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#56

Post by evanweaver »

gigatexal wrote: post #50
evanweaver wrote: post #48

Earlier quoted context omitted.

This a design choice to guarantee predictable performance at scale, not an architectural limitation, and subject to revision. You don't want your service to collapse when the query optimizer suddenly doesn't choose the right index anymore. That was a hard lesson learned for us at Twitter.

I agree -- it burns us in the SQL world too. I was just seeing it from a developer cognitive load point of view and maybe as a premature optimization, again, though if you guys and gals put this in place because of war stories from twitter then it's not premature by any means. In the rare case when no index or the wrong index is used in a SQL query an index hint is all it takes to fix or updating statistics of a tabl…

I should also add that Fauna indexes are more akin to views. They can cover multiple source collections and transform the data as well. So it’s not always possible to detect which index is correct via heuristic.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#57

Post by gigatexal »

evanweaver wrote: post #56
gigatexal wrote: post #50

Earlier quoted context omitted.

I agree -- it burns us in the SQL world too. I was just seeing it from a developer cognitive load point of view and maybe as a premature optimization, again, though if you guys and gals put this in place because of war stories from twitter then it's not premature by any means. In the rare case when no index or the wrong index is used in a SQL query an index hint is all it takes to fix or updating statistics of a tabl…

I should also add that Fauna indexes are more akin to views. They can cover multiple source collections and transform the data as well. So it’s not always possible to detect which index is correct via heuristic.

Ahh makes sense. Interesting.

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#58

Post by mcguire »

jchanimal wrote: post #22
jamesblonde wrote: post #20

Earlier quoted context omitted.

You are correct that they serialize cross-partition transactions. So if your workload has a reasonable percentage of write/update/delete operations it is possible that you will bottleneck on the global coordinator. From this blog https://fauna.com/blog/distributed-acid-transaction-performa... , you can get 3300 transactions/second. Daniel Abadi claims you can get to 500,000 trans/sec - http://dbmsmusings.blogspot.com…

The log segments committed by FaunaDB contain batches of transactions, which means our throughput is constrained not by our consensus protocol, but rather by conflicts as transactions are resolved. The benchmarked 3300 transactions/second mentioned is for complex transactions with dozens of reads and writes. Additionally, read-only transactions are not run through the transaction pipeline, since they can be served co…

Can you partition your batch transactions, so that all up to the conflict succeed?

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#59

Post by andydb »

The paper contains this claim:

> FaunaDB is an elegant, software-only solution for achieving global ACID transactions, with complete guarantees of serializability and consistency.

The paper makes it sound like FaunaDB claims strict serializability for all transactions (like Spanner). This means that if Txn(A) ends before Txn(B) begins, then Txn(B) must be guaranteed to see all data written by Txn(A).

However, what if Txn(B) is a read-only transaction? According to the paper, read-only transactions do not contact the global sequencer. Here is an example where an application would fail to read its own writes:

1. Txn(A) performs a write W(X) of data on replica R1.

2. Txn(A) is assigned a logical timestamp of T1 by the global log.

3. Conflict verification is triggered. Replica R1 plays back the global log and verifies there is no conflict.

4. Txn(A) completes, since one replica has verified there are no conflicts, and passes control back to the application.

5. The application now triggers a read-only Txn(B).

6. The coordinator for Txn(B) decides to read replica R2 rather than R1. However, the coordinator is not yet aware that the global log is now at timestamp T1, and it picks T0 as its snapshot timestamp.

7. It reads the value of X on replica R2, which does not yet reflect W(X).

-- From the application's point of view, it did not read its own writes. --

I'm not familiar enough with Fauna DB's subtleties to know if this scenario is possible in practice. Perhaps you could comment?

I did notice that at the end of the article, the language is carefully phrased to only make the claim of "serializable" (not "strictly serializable") for read-only transactions. But that would fall short of the guarantees that Spanner and Calvin make, and undermine the "complete guarantees of serializability and consistency".

Re: Consistency Without Clocks: FaunaDB's Distributed Transaction Protocol

#60

Post by voidmain »

evanweaver wrote: post #37
voidmain wrote: post #26

Earlier quoted context omitted.

FoundationDB has long supported both synchronous and asynchronous replication across regions, and its major users use multi-regional configurations. In the former mode, you will see 1xRTT latency for commits [1] from the active datacenter and 2xRTT latency for commits from other datacenters. In the latter mode, commits from the active datacenter are fast (0xRTT) but the durability property must be sacrificed if a reg…

Very interesting. People really care about latency so we are also look at more datacenter-local journaling schemes that maintain consistency at the expense of a theoretically unlikely hit to durability. What do you mean, "active datacenter"? Can all datacenters accept transactions concurrently?

If you are using Raft for replication, at any given time your replica set has a leader and it is located somewhere, and I would assume that writes from near there are faster than from anywhere else. In FoundationDB this is handled at a somewhat higher layer of the system, and since our transaction pipeline is (very roughly) resolve conflicts -> transaction log -> storage rather than transaction log -> resolve conflicts -> storage, we are also doing conflict resolution in that region.

Moreover, most current users of FoundationDB aren't willing to accept even 1xRTT latencies anywhere in their transaction lifecycle, so they can't abstract away which region is the fast one. A common way (though not the only way) to set things up is that your whole application (not just the database) is active in region A, but prepared to fail over at a moment's notice to run in region B. Ideally this comes at basically no performance penalty relative to a single-region setup in region A. Alternatively, it's possible to read or write the database from passive region(s) (and individual reads will normally happen from the local region with low latency), but each transaction has to pay a geographic latency at some point (or, in the case of read-only transactions, accept reduced consistency).

I think it would be possible to implement something analogous to satellite replication for Raft. It's a really nice set of tradeoffs for many applications, and if your cloud provider or equivalent has their act at all together instantaneous failures of all the datacenters in a region or their networking should really be pretty rare.

Post reply on HN