Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

151–160 of 453 posts

Re: Apple open-sources FoundationDB

#151

Post by voidmain »

jamesblonde wrote: post #121
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have: row-level locks partition-pruned index scans non-serialized cross-partition transactions (that is, a transaction coordi…

It's somewhat hard to answer your questions because the architecture (and hence, terminology) of FoundationDB is a little different than I think you are used to. But I will give it a shot.

FoundationDB uses optimistic concurrency, so "conflict ranges" rather than "locks". Each range is a (lexicographic) interval of one or more keys read or written by a transaction. The minimum granularity is a single key.

FoundationDB doesn't have a feature for indexing per se at all. Instead indexes are represented directly in the key/value store and kept consistent with the data through transactions. The scalability of this approach is great, because index queries never have to be broadcast to all nodes, they just go to where the relevant part of the index is stored.

FoundationDB delivers serializable isolation and external consistency for all transactions. There's nothing particularly special about transactions that are "cross-partition"; because of our approach to indexing and data model design generally we expect the vast majority of transactions to be in that category. So rather than make single-partition transactions fast and everything else very slow, we focused on making the general case as performant as possible.

Transaction coordination is pretty different in FoundationDB than in 2PC-based systems. The job of determining which conflict ranges intersect is done by a set of internal microservices called "resolvers", which partition up the keyspace totally independently of the way it is partitioned for data storage.

Please tell me if that leaves questions unresolved for you!

Re: Apple open-sources FoundationDB

#152

Post by segmondy »

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Or perhaps it's not so incredible? Maybe it wasn't such a huge hit for Apple and didn't leave up to expectation so they figure they can give it away and earn some community goodwill.

Re: Apple open-sources FoundationDB

#153

Post by voidmain »

jtchang wrote: post #131
wwilson wrote: post #98

Earlier quoted context omitted.

A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos). In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.

What if the number of boxes are even? Or do you get around this by not deploying an uneven number of boxes?

The number of coordinators is separate from the number of boxes. You don't have to have a coordinator on every box.

I think you can set the number of coordinators to be even, but you never should - the fault tolerance will be strictly better if you decrease it by one.

Re: Apple open-sources FoundationDB

#154

Post by wll »

voidmain wrote: post #126
wll wrote: post #112

Earlier quoted context omitted.

The datacenter-aware mode documentation [0] says “Although data will always be triple replicated in this mode, it may not be replicated across all datacenters.” Why is that? [0] https://apple.github.io/foundationdb/configuration.html?data...

I think it's just saying that it's willing to place two of the three replicas in a datacenter, for example if one of the three datacenters is down. This has downsides, since losing a datacenter will make it aggressively fill up disks, but mitigates against subsequent failures causing data loss. Most of the people who have run FoundationDB at scale have, for performance reasons, used configurations other than the "dat…

How would FoundationDB stay externally consistent with asynchronous cross-region replication?

Thank you for your time and FoundationDB—along with @nlavezzo, and team(s)!

Re: Apple open-sources FoundationDB

#156

Post by kuschku »

zzzcpan wrote: post #80
misterbowfinger wrote: post #43

I hate to be that person, but when I hear "ACID transactions in a distributed database", I hear Citus/Spanner/CockroachDB. I'm positive that Citus & Spanner are quite different from FoundationDB, but I have no idea how. Googling didn't help much. Can someone provide an overview of the differences?

It's more like what CockroachDB or TiDB use underneath. They all are suited for local clusters, but cannot perform well enough replicating across multiple far away datacenters over public internet, latency trade offs would be unbearable. Spanner is a bit different, with Google's fancy clocks and fancy networks, it can get better latency trade offs that might satisfy more applications. Citus I can't remember, but if C…

CockroachDB (and I assume TiDB as well) can have the same properties and same awesome latencies as Spanner, if you have the same hardware.

So if you actually put down your own fiber, and install GPS clocks in each rack, you’ll be able to enjoy the same results.

Re: Apple open-sources FoundationDB

#157

Post by infogulch »

voidmain wrote: post #129
infogulch wrote: post #119

Earlier quoted context omitted.

> It can't tolerate N failures from N+1 copies of your data Sorry I got the terminology wrong, but that's a distinction without a difference. If it can tolerate N failures from N+1 copies, that means a network partition would allow any one copy to continue chugging along making changes by itself. You have two options: consistency is dropped and you downgrade to eventually consistent (at best), or availability is drop…

FoundationDB stores 2N+1 copies of some "coordination state" and does a consensus algorithm whenever it is updated. But this state doesn't contain a copy of your data; basically think of it as storing a replication configuration. It's very small and rarely changes. In the happy case, replication takes place using the replicas and quorum rules specified by this configuration. For example, you might require writes to s…

Nope your explanation made sense, thank you!

When I wrote that I was wondering if it used a second 2N+1 dataset just for coordination & consensus. This has the benefit of separating data from consensus, allowing the N of N+1 data failure. But at the end of the day consistency still comes down to a N of 2N+1 failure tolerance of that second coordination state. It's smaller easier to replicate etc etc but it seems like it still has the same fault tolerance as just replicating the data 2N+1 times. It sounds like it's worked out great in practice for FDB.

But you say it rarely changes... but wouldn't it have to change every time there's a change to the dataset? I feel like this means you have to do even more replication and consensus than just replicating the data without this second consensus state.

Re: Apple open-sources FoundationDB

#158

Post by tjb1982 »

wwilson wrote: post #105
X-Istence wrote: post #102

Earlier quoted context omitted.

Is the wire-compatible clone of the MongoDB API available?

It doesn't look like Apple open-sourced the Document Layer, which is a slight bummer. But I echo what Dave said below: what we got is incredible, let's not get greedy! Also TBH now that I don't have commercial reasons to push interop, if I write another document database on top of FDB, I doubt I'd make it Mongo compatible. That API is gnarly.

I was pretty bummed not see that, myself

Re: Apple open-sources FoundationDB

#159

Post by chx »

wwilson wrote: post #105
X-Istence wrote: post #102

Earlier quoted context omitted.

Is the wire-compatible clone of the MongoDB API available?

It doesn't look like Apple open-sourced the Document Layer, which is a slight bummer. But I echo what Dave said below: what we got is incredible, let's not get greedy! Also TBH now that I don't have commercial reasons to push interop, if I write another document database on top of FDB, I doubt I'd make it Mongo compatible. That API is gnarly.

> That API is gnarly.

Out of the many MongoDB criticisms, this one is valid. This https://www.linkedin.com/pulse/mongodb-frankenstein-monster-... article is quite right about it (note: endorsing the article does not mean I endorse its author by far).

Other than that, they totally did a fake it until you make it with MongoDB 3.4 passing Jepsen a year ago and MongoDB BI 2.0 containing their own SQL engine instead of wrapping PostgreSQL.

Re: Apple open-sources FoundationDB

#160

Post by tjb1982 »

rthomas900 wrote: post #137
wwilson wrote: post #60

Earlier quoted context omitted.

As an existence proof: before the acquisition we built an ANSI SQL database and a wire-compatible clone of the MongoDB API. I see no reason you wouldn't be able to implement Datastore. In fact here's a public source claiming that Firestore (which I believe is its successor) is implemented on top of Spanner: https://www.theregister.co.uk/2017/10/04/google_backs_up_fir...

If I recall correctly, the "SQL layer" you had in FDB before the Apple acquisition was a nice proof of concept, but lacked support for many features (joins in SELECT, for example). Is the SQL layer code from that time available anywhere to the public? (I'm not seeing it in the repo announced by OP.)

I used to work there. The SQL layer was capable of actually the majority of SQL features including joins, etc. We had an internal rails app that we used to dog-food for performance monitoring, etc. I used to work on the document layer, and was sad to see it wasn't included here.
Post reply on HN