Live data from Hacker News

120K distributed consistent writes per second with Calvin

fauna.com

21–30 of 73 posts

Re: 120K distributed consistent writes per second with Calvin

#21
post #13
post #8

Earlier quoted context omitted.

Calvin is a generalized consistency protocol, that we use in FaunaDB to support relational semantics (but not SQL) in our database. Multi-query transactions can be useful, but the FaunaDB query language is functional, rather than declarative like SQL, so composing queries that can do everything you want is usually easier than SQL.

How would you perform the classic "transfer money from one account to another" operation? Would you create a single operation that reads one record, checks that it's enough, then adds the amount to another record? Or maybe you'd first read both accounts, then issue a conditional write operation that makes sure the data hasn't changed before doing the write?

FaunaDB's query language makes it straightforward to do it the first way. All queries are serializable, so any preconditions checked would gate transaction commit as you would expect, and read-modify-write style transactions work.

edit: here's an example in our Scala DSL:

  Let {
    val amount = 50
    val balanceA = Select("data" / "balance", Get(Ref("accountA"))
    val balanceB = Select("data" / "balance", Get(Ref("accountB"))
    If(Gteq(balanceA, amount),
      Do(
        Update(Ref("accountA"), Obj("data" -> Obj("balance" -> Subtract(balanceA, amount)))),
        Update(Ref("accountB"), Obj("data" -> Obj("balance" -> Add(balanceB, amount)))),
        "Transfer Success"
      ),
      "Insufficient Funds"
    )
  }

Re: 120K distributed consistent writes per second with Calvin

#24
post #19
post #12

Earlier quoted context omitted.

Unfortunately I'm the last person to ask. While I did start at FoundationDB pretty early (second employee), I ceased to be involved at the point of the acquisition, and beyond that I've only heard a few rumors from former coworkers. As a business it was always an ambitious effort, and I'm not sure what could or should have been done differently. But since then I've used a number of other systems and thought to myself…

Another former-FoundationDB guy here (hi Ian!), and I actually think the business case for Apple open sourcing is very strong. I'm a fan of the layered architecture we chose, but building efficient and powerful layers on top of the core key-value store is a serious engineering effort in its own right. By encouraging an open-source layer ecosystem (and operational and deployment tools), Apple could leverage its invest…

We specifically chose a monolithic architecture for FaunaDB, since performance improvements invariably come from breaking interface boundaries and sharing additional information. It's been working out well.

Re: 120K distributed consistent writes per second with Calvin

#25
post #23

Earlier quoted context omitted.

They are durable; will clarify.

Otherwise it would be like Mongo did - we have put your data in the OS buffers - what could possibly go wrong?

Haha no...it's the opposite of that.

Re: 120K distributed consistent writes per second with Calvin

#26
This description is very misleading.

120,000 writes per second is accurate, talking about actual durable storage (disk) writes. But it's only 3,330 transactions, which should be the number that a user cares about.

I don't have proper data and I'm a bit rusty, but I feel like Cassandra could blow that away if you set similar consistency requirements on the client side (QUORUM on read, same for write?). Am I understanding this correctly, or does Fauna/Calvin give you something functionally better than what C* can do?

Re: 120K distributed consistent writes per second with Calvin

#27
post #19

Earlier quoted context omitted.

Another former-FoundationDB guy here (hi Ian!), and I actually think the business case for Apple open sourcing is very strong. I'm a fan of the layered architecture we chose, but building efficient and powerful layers on top of the core key-value store is a serious engineering effort in its own right. By encouraging an open-source layer ecosystem (and operational and deployment tools), Apple could leverage its invest…

We specifically chose a monolithic architecture for FaunaDB, since performance improvements invariably come from breaking interface boundaries and sharing additional information. It's been working out well.

Yes, this is the argument that VoltDB made as well:

https://www.voltdb.com/blog/foundationdbs-lesson-fast-key-va...

My feelings on this topic are mixed. On the one hand, I think many of the specific examples chosen in that post are false (and have told John as much in person). On the other hand, the general point that you can squeeze out constant factor performance improvements by violating abstraction boundaries is obviously usually true.

Nevertheless, I still think this is a bad argument. While it's true that abstractions are rarely costless, they can often be made so cheap that the low-hanging performance fruit is elsewhere. And in particular, cheap enough that they're worth it when you consider all the other benefits that they bring.

When I built a query language and optimizer on top of FoundationDB, my inability to push type information down into the storage engine was about the last thing on my mind. Perhaps someday when I'd made everything else perfect it would've become a big pain (and perhaps someday we would've provided more mechanisms for piercing the various abstractions and providing workload hints), but in the meantime partitioning off all state in the system into a dedicated component that handled it extremely well made the combined distributed systems problem massively more tractable. The increased developer velocity and reduced bugginess in turn meant that I (as a user of the key-value store) could spend scarce engineering resources on other sorts of performance improvements that more than compensated for the theoretical overhead imposed by the abstraction.

I won't claim that a transactional ordered key-value store is the perfect database abstraction for every situation, but it's one that I've found myself missing a great deal since leaving Apple.

But I'm glad to hear that things are going well for you guys. Best of luck, this is a brutal business!

Re: 120K distributed consistent writes per second with Calvin

#30
post #26

This description is very misleading. 120,000 writes per second is accurate, talking about actual durable storage (disk) writes. But it's only 3,330 transactions, which should be the number that a user cares about. I don't have proper data and I'm a bit rusty, but I feel like Cassandra could blow that away if you set similar consistency requirements on the client side (QUORUM on read, same for write?). Am I understand…

A more apples-to-apples comparison with Cassandra would be FaunaDB transactions and Cassandra's atomic batch mutations, or its PAXOS-based lightweight transactions as opposed to single-cell writes tested in most Cassandra benchmarks.

YMMV, but we've found the performance of Cassandra writing out similar-sized multi-row atomic batches at QUORUM to be similar in this hardware configuration.

FaunaDB transactions are quite a bit more powerful, as they can span multiple keys, use conditionals and read-modify-write logic, and still resolve with serializable semantics.

Post reply on HN