Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

120K distributed consistent writes per second with Calvin

fauna.com

Post reply on HN
Print view View original

41–50 of 73 posts

Re: 120K distributed consistent writes per second with Calvin

#41

Post by otterley »

qaq wrote: post #39

Maybe I am missing some special point but a decent PG box will do 1,000,000+ TPS vs 3,000+ TPS here. When pgXact lands it will do close to 2,000,000 TPS. So reading all the posts about the amazing new db "X" that can do about N times less than PG on a multi-node cluster I get confused why the numbers are being presented as some sort of achievement.

That doesn't sound right. Even the newest NVMe devices can't do 1M writes per second; they're maxing out at around 330k IOPS.

The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar...). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.

Re: 120K distributed consistent writes per second with Calvin

#42

Post by qaq »

otterley wrote: post #41
qaq wrote: post #39

Maybe I am missing some special point but a decent PG box will do 1,000,000+ TPS vs 3,000+ TPS here. When pgXact lands it will do close to 2,000,000 TPS. So reading all the posts about the amazing new db "X" that can do about N times less than PG on a multi-node cluster I get confused why the numbers are being presented as some sort of achievement.

That doesn't sound right. Even the newest NVMe devices can't do 1M writes per second; they're maxing out at around 330k IOPS. The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar... ). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.

330K iops for a single device you are very unlikely to be running a single device. There are Fusion IO models that can do 1M IOPS but they are on the exotic side. If you are optimising for throughput you can configure commit_delay so you will fsync multiple commits.

Re: 120K distributed consistent writes per second with Calvin

#43

Post by [deleted] »

qaq wrote: post #42
otterley wrote: post #41

Earlier quoted context omitted.

That doesn't sound right. Even the newest NVMe devices can't do 1M writes per second; they're maxing out at around 330k IOPS. The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar... ). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.

330K iops for a single device you are very unlikely to be running a single device. There are Fusion IO models that can do 1M IOPS but they are on the exotic side. If you are optimising for throughput you can configure commit_delay so you will fsync multiple commits.

[deleted]

Re: 120K distributed consistent writes per second with Calvin

#44

Post by abdullin »

wwilson wrote: post #19
itp wrote: post #12

Earlier quoted context omitted.

Unfortunately I'm the last person to ask. While I did start at FoundationDB pretty early (second employee), I ceased to be involved at the point of the acquisition, and beyond that I've only heard a few rumors from former coworkers. As a business it was always an ambitious effort, and I'm not sure what could or should have been done differently. But since then I've used a number of other systems and thought to myself…

Another former-FoundationDB guy here (hi Ian!), and I actually think the business case for Apple open sourcing is very strong. I'm a fan of the layered architecture we chose, but building efficient and powerful layers on top of the core key-value store is a serious engineering effort in its own right. By encouraging an open-source layer ecosystem (and operational and deployment tools), Apple could leverage its invest…

Layer modeling techniques shared by FoundationDB (and still available in the internet archive) are still immensely helpful even without the database.

We are happily using them to implement and optimize our local storage on top of LMDB (another awesome database). However, these approaches could be applied to any other key-value database with transactions and lexicographically stored keys.

Re: 120K distributed consistent writes per second with Calvin

#45

Post by abdullin »

wwilson wrote: post #27
evanweaver wrote: post #24

Earlier quoted context omitted.

We specifically chose a monolithic architecture for FaunaDB, since performance improvements invariably come from breaking interface boundaries and sharing additional information. It's been working out well.

Yes, this is the argument that VoltDB made as well: https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va... My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is…

Monolithic databases (CockroachDB, Spanner etc) don't eliminate abstraction boundaries on the larger scale. They are simply aligned with the database boundary, pushing the burden of crossing them to the application logic. Software will still have to pay the price of crossing the gap with code complexity and performance (object-relational impedance mismatch comes to my mind).

It feels like the building block approach lets you achieve better design and performance for your entire application in the long run. Especially, if you can treat building blocks as blueprints and modify them to fit the task at hand.

Re: 120K distributed consistent writes per second with Calvin

#46

Post by evanweaver »

qaq wrote: post #42
otterley wrote: post #41

Earlier quoted context omitted.

That doesn't sound right. Even the newest NVMe devices can't do 1M writes per second; they're maxing out at around 330k IOPS. The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar... ). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.

330K iops for a single device you are very unlikely to be running a single device. There are Fusion IO models that can do 1M IOPS but they are on the exotic side. If you are optimising for throughput you can configure commit_delay so you will fsync multiple commits.

FaunaDB is faster in an unclustered, unreplicated configuration too, but that's not what FaunaDB is really for.

We should track disk IOPS, though, so you can do an apples-to-apples comparison of how much low-level throughput the database is driving. I believe the instance store disks in the EC2 C3 hardware class can support about ~20K write IOPS each.

Re: 120K distributed consistent writes per second with Calvin

#47

Post by g0del_was_wr0ng »

qaq wrote: post #39

Maybe I am missing some special point but a decent PG box will do 1,000,000+ TPS vs 3,000+ TPS here. When pgXact lands it will do close to 2,000,000 TPS. So reading all the posts about the amazing new db "X" that can do about N times less than PG on a multi-node cluster I get confused why the numbers are being presented as some sort of achievement.

I agree that your PG numbers don't sound likely/factual -- the reason for your confusion is probably that somebody gave you untrue performance numbers for postgres or you're not comparing the same things. Is the 1m+ TPS something you measured yourself or "heard from a friend"?

If you ran the benchmark yourself, how did you achieve 1m durable writes/sec on a postgres machine/instance? [It's quite an achievement] On what kind of crazy hardware? How large was each write/row? Did you use the postgres network protocol to perform the writes?

Re: 120K distributed consistent writes per second with Calvin

#48

Post by g0del_was_wr0ng »

Including your 9x write amplification in the number of "consistent writes" doesn't count -- like at all. I'm amazed nobody called you out on this yet.

You're doing 3k batches per second with 4 logical writes each, right? So that is at most 3-12k writes per second using the way that every other distributed database benchmark and paper counts.

Or otherwise - if you continue counting writes in this special/misleading way - you'd have to multiply every other distributed db benchmark's performance numbers with a factor of 3-15x to get an apples-to-apples comparison.

The 12k batched writes/sek through what I assume is a paxos variant is still pretty impressive though! Good to get more competition/alternatives for zookeeper & friends!

Re: 120K distributed consistent writes per second with Calvin

#49

Post by pg314 »

qaq wrote: post #39

Maybe I am missing some special point but a decent PG box will do 1,000,000+ TPS vs 3,000+ TPS here. When pgXact lands it will do close to 2,000,000 TPS. So reading all the posts about the amazing new db "X" that can do about N times less than PG on a multi-node cluster I get confused why the numbers are being presented as some sort of achievement.

Like others have commented, those numbers seem to be too high for writes.

On the other hand, the Fauna numbers don't seem that impressive to me. On a mid-2011 Macbook Air, I get 2600 transactions per second (read-committed) in PostgreSQL 9.6. Setup is as follows:

  CREATE TABLE IF NOT EXISTS foo(a TEXT, b TEXT, c TEXT, d TEXT);
  CREATE INDEX IF NOT EXISTS idx_foo_a ON foo(a);
  CREATE INDEX IF NOT EXISTS idx_foo_b ON foo(b);
  CREATE INDEX IF NOT EXISTS idx_foo_c ON foo(c);
  CREATE INDEX IF NOT EXISTS idx_foo_d ON foo(d);

  -- prepared statement, the inserted strings are 4 chars wide
  INSERT INTO foo(a, b, c, d)
  VALUES
    ($1, $2, $3, $4),
    ($5, $6, $7, $8),
    ($9, $10, $11, $12),
    ($13, $14, $15, $16);
These numbers are for one thread doing the writing.

Am I missing something?

Re: 120K distributed consistent writes per second with Calvin

#50

Post by chillydawg »

qaq wrote: post #42
otterley wrote: post #41

Earlier quoted context omitted.

That doesn't sound right. Even the newest NVMe devices can't do 1M writes per second; they're maxing out at around 330k IOPS. The 1M "TPS" you're referring to is a read-only benchmark (e.g. http://akorotkov.github.io/blog/2016/05/09/scalability-towar... ). Those are reads (most likely from the buffer cache), not writes or transactions in any real sense.

330K iops for a single device you are very unlikely to be running a single device. There are Fusion IO models that can do 1M IOPS but they are on the exotic side. If you are optimising for throughput you can configure commit_delay so you will fsync multiple commits.

Relaxing disk commits is a quick route to data loss. Might as well use mongo at that point.
Post reply on HN