Apple open-sources FoundationDB • Hacker News Board

Re: Apple open-sources FoundationDB

#411

Post by voidmain » Sat, Apr 21, 2018, 2:08 PM UTC

voidmain wrote: post #212

wll wrote: post #202

Earlier quoted context omitted.

> So why not go active/passive, and have at least one of your datacenters be fast? While local writes would stay fast, wouldn’t active/passive see higher-latency non-local writes than Spanner or Fauna’s (assuming a NAM-EUR-ASIA topology)? I agree with and do appreciate the multiple FoundationDB clusters suggestion.

I'm speculating, but I think in this mode, from the "slow" datacenters you would see one round trip time to start a transaction, then reads will be fast (they can be done safely from your local datacenter because of MVCC), and then one round trip time to commit the transaction. I think that's as good as Spanner does with the same geography, but I'm not sure. I think you could get rid of the first round trip time even…

Update: Apparently Spanner is way slower than I thought in "slow" datacenters, doing a round trip for every transactional read. So this would absolutely stomp that. (Although spanner, as a higher level system, has the ability to move different data to be fast in different regions built in, which is nice, and as I said it will probably have to be a layer feature in the fdb world)

Re: Apple open-sources FoundationDB

#412

Post by polskibus » Sat, Apr 21, 2018, 2:38 PM UTC

voidmain wrote: post #410

polskibus wrote: post #405

Earlier quoted context omitted.

That would mean that SSI is not used to provide Serializable isolation. level. If so, what is used instead? 2 phase locking? I thought it's not very scalable ?

I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950 I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordan…

Thanks, I'd love to have a bit more of your attention, foundationdb seems very interesting but I need to know a bit more :)

Let me expand the definition in Kleppmann's book then.I think it is important because it creates a difference between SSI and typical Serializable level based on 2PL. The below is paraphrasing the definitions on p. 324-329. The book references http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf. (I must admit, I read the book, not the paper).

Basic idea - make a system appear as if there were only one copy of the data and ALL operations on it are atomic. In this model, there may be replicas, but we don't care about them. As soon as a client completes a write to the db, all clients reading the db must be able to see the value just written.

In SSI this is not true, because you may the snapshot may not include writes more recent than the snapshot -> reads from the snapshot are not lineraizable.

Linearizable CAS register is equivalent to consensus, and can provide total order. It is therefore what most developers would love to have (if cost was not an issue :) )

Re: Apple open-sources FoundationDB

#413

Post by Joeri » Sat, Apr 21, 2018, 2:41 PM UTC

qaq wrote: post #201

panghy wrote: post #182

Earlier quoted context omitted.

We basically replaced mySQL, Zookeeper and HBase with a single KV store that supports transactions, watches, and scales. It's not a trivial point that you can just develop code against a single API (finally Java 8 CompletableFutures) and not have to set up a ton of dependencies when you are building on top of FDB. We are (obviously) experts at monitoring FoundationDB with Wavefront and we hope to release the metric h…

"but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS " <=== This I wish more people realized that is a day to day reality if you are in AWS at scale.

The best way I've heard it described is "complex systems run in degraded mode".

https://cdn.chrisshort.net/How-Complex-Systems-Fail.pdf

Basically once a system is complex enough some part if it is always broken. The software must be designed from the assumption that the system is never running flawlessly.

Re: Apple open-sources FoundationDB

#414

Post by rapsey » Sat, Apr 21, 2018, 2:50 PM UTC

thx4thefish wrote: post #267

This seems nice. Besides a bunch of fanboy comments coming from the creators and devs, why is this exciting to the rest of us where things like the capability to join tables in an rdbms is trivial.

FDB is lower layer. It makes it possible to build a distributed rdbms on top of it. There are is some competition in this area, nothing remotely as mature as FDB.

Re: Apple open-sources FoundationDB

#415

Post by polskibus » Sat, Apr 21, 2018, 4:18 PM UTC

voidmain wrote: post #410

polskibus wrote: post #405

Earlier quoted context omitted.

That would mean that SSI is not used to provide Serializable isolation. level. If so, what is used instead? 2 phase locking? I thought it's not very scalable ?

I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950 I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordan…

as far as I know SSI as implemented in Postgresql, aborts the way you describe, as per https://drkp.net/papers/ssi-vldb12.pdf

Re: Apple open-sources FoundationDB

#416

Post by voidmain » Sat, Apr 21, 2018, 5:01 PM UTC

polskibus wrote: post #412

voidmain wrote: post #410

Earlier quoted context omitted.

I explain the basics of our concurrency control here: https://news.ycombinator.com/item?id=16877950 I guess textbook SSI is willing to "reorder" conflicting transactions if the result is still serializable, which could violate external consistency if you don't have any other bounds on the order. In the language of SSI, fdb simply aborts the later of any pair of read/write transactions with an rw-conflict, in accordan…

Thanks, I'd love to have a bit more of your attention, foundationdb seems very interesting but I need to know a bit more :) Let me expand the definition in Kleppmann's book then.I think it is important because it creates a difference between SSI and typical Serializable level based on 2PL. The below is paraphrasing the definitions on p. 324-329. The book references http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf…

From the paper you link:

"A history is serializable if it is equivalent to one in which transactions appear to execute sequentially, i.e., without interleaving... A history is strictly serializable if the transactions’ order in the sequential history is compatible with their precedence order... Linearizability can be viewed as a special case of strict serializability where transactions are restricted to consist of a single operation applied to a single object."

In these terms, FoundationDB has the strict serializability property, and thus if you do exactly one operation in each FoundationDB transaction then that is linearizable.

But that kind of linearizability is much less powerful than what FoundationDB actually gives you. You cannot efficiently maintain global invariants, like indexes, with single-operational linearizability. I don't think this definition is very useful! I think strict serializability (which is to say serializability & external consistency) is what you actually want.

A linearizable CAS register can be implemented in FDB as simply as this:

  @fdb.transactional
  def compare_and_set( tr, key, vold, vnew ):
    if tr[key] == vold:
      tr[key] = vnew

but this is not the limit of what you can do.

Re: Apple open-sources FoundationDB

#417

Post by srigi » Sat, Apr 21, 2018, 5:22 PM UTC

bsaul wrote: post #6

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Your talk was one of the best talk i've seen , and i keep mentionning it to people whenever they ask me about distributed systems, database and testing. i'm incredibly impatient to have a look at what the community is going to build on top of that very promising technology.

Can you share that talk here too?

Re: Apple open-sources FoundationDB

#418

Post by polskibus » Sat, Apr 21, 2018, 6:29 PM UTC

voidmain wrote: post #416

polskibus wrote: post #412

Earlier quoted context omitted.

Thanks, I'd love to have a bit more of your attention, foundationdb seems very interesting but I need to know a bit more :) Let me expand the definition in Kleppmann's book then.I think it is important because it creates a difference between SSI and typical Serializable level based on 2PL. The below is paraphrasing the definitions on p. 324-329. The book references http://cs.brown.edu/~mph/HerlihyW90/p463-herlihy.pdf…

From the paper you link: "A history is serializable if it is equivalent to one in which transactions appear to execute sequentially, i.e., without interleaving... A history is strictly serializable if the transactions’ order in the sequential history is compatible with their precedence order... Linearizability can be viewed as a special case of strict serializability where transactions are restricted to consist of a…

Thank you very much for your in-depth explanation, I believe the only thing left for me is to run FDB myself, sounds very promising :) FDB replacing zookeper + sth else would reduce the complexity of target distributed system, almost too good to be true.

Re: Apple open-sources FoundationDB

#419

Post by banana-guy » Sun, Apr 22, 2018, 2:14 AM UTC

segmondy wrote: post #152

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Or perhaps it's not so incredible? Maybe it wasn't such a huge hit for Apple and didn't leave up to expectation so they figure they can give it away and earn some community goodwill.

there were rumors that fdb had never been adopted at Apple because of some limitations and they still ran an in-house version of cassandra

Re: Apple open-sources FoundationDB

#420

Post by truth_seeker » Sun, Apr 22, 2018, 6:29 AM UTC

I think Apache Ignite distributed KV store is much better choice as it has 2PC, indexes, distributed computing idioms and can be embedded as library in JVM app. Plus it supports SQL engine, thanks to H2 SQL parser engine.

You can also create graph layer over it using gremlin in a day or two.