Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

120K distributed consistent writes per second with Calvin

fauna.com

Post reply on HN
Print view View original

31–40 of 73 posts

Re: 120K distributed consistent writes per second with Calvin

#31

Post by atombender »

itp wrote: post #12
halestock wrote: post #11

Earlier quoted context omitted.

Bah! I was so dissappointed when I heard about the Apple acquisition of FoundationDB. Will any of the technology behind it ever see the light of day?

Unfortunately I'm the last person to ask. While I did start at FoundationDB pretty early (second employee), I ceased to be involved at the point of the acquisition, and beyond that I've only heard a few rumors from former coworkers. As a business it was always an ambitious effort, and I'm not sure what could or should have been done differently. But since then I've used a number of other systems and thought to myself…

What do you guys think about TiDB and CockroachDB, both of which are SQL layers on top of a distributed K/V store?

Re: 120K distributed consistent writes per second with Calvin

#32

Post by rystsov »

jchrisa wrote: post #29
rystsov wrote: post #28

Is it possible to download fauna to play with it on my own?

You can sign up for the cloud version in a few seconds here: https://fauna.com/serverless-cloud-sign-up

With the cloud version, it's impossible to run jepsen-like tests to validate consistency and to observe cluster's behavior when the network is unstable and nodes tend to crush.

Re: 120K distributed consistent writes per second with Calvin

#33

Post by wwilson »

atombender wrote: post #31
itp wrote: post #12

Earlier quoted context omitted.

Unfortunately I'm the last person to ask. While I did start at FoundationDB pretty early (second employee), I ceased to be involved at the point of the acquisition, and beyond that I've only heard a few rumors from former coworkers. As a business it was always an ambitious effort, and I'm not sure what could or should have been done differently. But since then I've used a number of other systems and thought to myself…

What do you guys think about TiDB and CockroachDB, both of which are SQL layers on top of a distributed K/V store?

Full disclosure: I now work at Google on Cloud Spanner which competes with both products you mentioned. These are just my personal (and probably highly biased) opinions.

I have some concerns about CockroachDB on both the performance and the reliability fronts. But I hugely admire what they're trying to do and I've heard that they're rapidly improving in both areas. TiDB is an exciting project that I've heard great things about but have never tried myself. I think it's also relatively immature.

Honestly if I were starting a project right now and had neither FDB nor Spanner available to me, I'd probably try to push Postgres as far as I possibly could before considering anything else.

Re: 120K distributed consistent writes per second with Calvin

#34

Post by evanweaver »

wwilson wrote: post #27
evanweaver wrote: post #24

Earlier quoted context omitted.

We specifically chose a monolithic architecture for FaunaDB, since performance improvements invariably come from breaking interface boundaries and sharing additional information. It's been working out well.

Yes, this is the argument that VoltDB made as well: https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va... My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is…

"When I built a query language and optimizer on top of FoundationDB, my inability to push type information down into the storage engine was about the last thing on my mind."

What was on your mind? What performance problems did you encounter?

Re: 120K distributed consistent writes per second with Calvin

#35

Post by jchrisa »

rystsov wrote: post #32
jchrisa wrote: post #29

Earlier quoted context omitted.

You can sign up for the cloud version in a few seconds here: https://fauna.com/serverless-cloud-sign-up

With the cloud version, it's impossible to run jepsen-like tests to validate consistency and to observe cluster's behavior when the network is unstable and nodes tend to crush.

We have some correctness results coming that should address your concerns. Watch this space for updates.

Re: 120K distributed consistent writes per second with Calvin

#36

Post by olegkikin »

2011: Benchmarking Cassandra Scalability on AWS - Over a million writes per second

http://techblog.netflix.com/2011/11/benchmarking-cassandra-s...

Also a single SSD from 2015 is rated at 120K writes per second:

PM1725: http://www.samsung.com/semiconductor/global/file/insight/201...

Re: 120K distributed consistent writes per second with Calvin

#37

Post by web007 »

freels wrote: post #30
web007 wrote: post #26

This description is very misleading. 120,000 writes per second is accurate, talking about actual durable storage (disk) writes. But it's only 3,330 transactions, which should be the number that a user cares about. I don't have proper data and I'm a bit rusty, but I feel like Cassandra could blow that away if you set similar consistency requirements on the client side (QUORUM on read, same for write?). Am I understand…

A more apples-to-apples comparison with Cassandra would be FaunaDB transactions and Cassandra's atomic batch mutations, or its PAXOS-based lightweight transactions as opposed to single-cell writes tested in most Cassandra benchmarks. YMMV, but we've found the performance of Cassandra writing out similar-sized multi-row atomic batches at QUORUM to be similar in this hardware configuration. FaunaDB transactions are qui…

That makes a lot more sense then. It's still a misleading statement to say "writes" vs "transactions" since you could (potentially) make fewer writes and support more transactions. The ratio between the two is a measure of efficiency, but only transactions matter to end-users.

Re: 120K distributed consistent writes per second with Calvin

#38

Post by jhugg »

wwilson wrote: post #27
evanweaver wrote: post #24

Earlier quoted context omitted.

We specifically chose a monolithic architecture for FaunaDB, since performance improvements invariably come from breaking interface boundaries and sharing additional information. It's been working out well.

Yes, this is the argument that VoltDB made as well: https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va... My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is…

Hi Will. Thanks for the shout-out.

I still think many of the arguments in that blog post hold up for non-embedded KV stores. I think you can mitigate a lot by aggressively caching metadata, but eventually you end up moving the SQL engine closer and closer to the storage layer to get performance. And yeah, you end up more monolithic and testing gets harder. Sigh.

Some of this is workload dependent. If you're not touching many rows in your queries and transactions, then you can get away with a lot more. But if you give someone SQL, they're going to want to scan.

I wouldn't mind being proven wrong. Maybe Apple made FDB run SQL at legit speeds. I haven't seen much from public projects that work this way to change my mind yet.

> I won't claim that a transactional ordered key-value store is the perfect database abstraction for every situation, but it's one that I've found myself missing a great deal since leaving Apple.

How does Spanner not satisfy that itch? Not ordered matters?

Re: 120K distributed consistent writes per second with Calvin

#39

Post by qaq »

Maybe I am missing some special point but a decent PG box will do 1,000,000+ TPS vs 3,000+ TPS here. When pgXact lands it will do close to 2,000,000 TPS. So reading all the posts about the amazing new db "X" that can do about N times less than PG on a multi-node cluster I get confused why the numbers are being presented as some sort of achievement.

Re: 120K distributed consistent writes per second with Calvin

#40

Post by wwilson »

jhugg wrote: post #38
wwilson wrote: post #27

Earlier quoted context omitted.

Yes, this is the argument that VoltDB made as well: https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va... My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is…

Hi Will. Thanks for the shout-out. I still think many of the arguments in that blog post hold up for non-embedded KV stores. I think you can mitigate a lot by aggressively caching metadata, but eventually you end up moving the SQL engine closer and closer to the storage layer to get performance. And yeah, you end up more monolithic and testing gets harder. Sigh. Some of this is workload dependent. If you're not touch…

> How does Spanner not satisfy that itch? Not ordered matters?

I was probably unclear in my previous comment. Spanner is great! (And Spanner is ordered). The particular aspect of FDB that I miss is what some of our old customers called "the bottom half of a database" or "a database construction kit". In fact FDB was an awesome modular building block for all kinds of distributed systems, not just databases. We hacked up prototypes for a whole bunch of these but sadly never got around to releasing them.

Spanner is a full-fledged enterprise grade database with opinions about your data model, query language, types, etc. For the vast majority of customers, that's much more useful than what FDB provided. But for me as somebody who enjoys kicking around silly new ideas for distributed systems, it's a bit less fun.

Post reply on HN