Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

121–130 of 453 posts

Re: Apple open-sources FoundationDB

#121

Post by jamesblonde »

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

I am one of the designers of probably the best known metadata storage engine for a distributed filesystem, hopsfs - www.hops.io. When I looked at FoundationDB before Apple bought you, you supported transactions - great. But we need much more to scale. Can you tell me which of the following you have: row-level locks partition-pruned index scans non-serialized cross-partition transactions (that is, a transaction coordinator per DB node) distribution-aware transactions (hints on which TC to start a transaction on)

The relative performance contribution of most of those features (cross-partition transactions not evaluated -it's a must) can be seen in our paper at Usenix Fast: https://www.usenix.org/system/files/conference/fast17/fast17...

Re: Apple open-sources FoundationDB

#122

Post by tjb1982 »

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

That is so exciting! Can't wait to poke around. I wonder if there's anything in there I might recognize

Re: Apple open-sources FoundationDB

#123

Post by Longhanks »

saagarjha wrote: post #11

It's under Apache 2.0 for those curious: https://github.com/apple/foundationdb/blob/master/LICENSE . Also, side note: it looks like this was a private GitHub repository for at least a couple months, since they have pull requests going back for at least that long. I find this interesting, since Apple normally "cleans up" history before open sourcing.

Swift, too, was published on GitHub including its whole version control history, dating back to 2010 :)

Re: Apple open-sources FoundationDB

#124

Post by panghy »

voidmain wrote: post #106
mzeier wrote: post #99

Earlier quoted context omitted.

Truthfully at Wavefront we've taken the json status directly into telegraf. Plus a bunch of python tooling to massage additional telemetry on a clusters health (coordinator reachability for example). Plus even more tooling (mostly Ansible) for managing large fleets.

I saw @spullara do pretty neat stuff with our log files in Wavefront. Will you guys think about open sourcing tooling? Apple is realistically never going to do that stuff.

Everything is fair game, being a monitoring company, we certainly will have first-class fdb support. We already have tons of workflows and templates.

Re: Apple open-sources FoundationDB

#125

Post by helper »

I'm very interested in hearing more about what running FoundationDB in production is like.

I believe that FoundationDB stores rows in lexicographical order by key. Other databases like Cassandra strongly push you toward not storing data this way as it can easily lead to hotspots in the cluster. How do you deploy a FoundationDB cluster without leading to hotspots, or perhaps what operational actions are available to rebalance data?

Re: Apple open-sources FoundationDB

#126

Post by voidmain »

wll wrote: post #112
voidmain wrote: post #72

Earlier quoted context omitted.

I think Citus is not really ACID. Spanner (and to an extent its less mature OSS descendants Cockroach and TiKV) has more comparable goals, but is fairly different architecturally. For example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance (even lots of databases with much weaker guarantees are in the latter category!), doesn't trust clocks at all, doesn't lose performance when…

The datacenter-aware mode documentation [0] says “Although data will always be triple replicated in this mode, it may not be replicated across all datacenters.” Why is that? [0] https://apple.github.io/foundationdb/configuration.html?data...

I think it's just saying that it's willing to place two of the three replicas in a datacenter, for example if one of the three datacenters is down. This has downsides, since losing a datacenter will make it aggressively fill up disks, but mitigates against subsequent failures causing data loss.

Most of the people who have run FoundationDB at scale have, for performance reasons, used configurations other than the "datacenter aware" mode for their inter region replication, so they may not be the strongest thing operationally.

There is some work that from what I can see in the code is still in progress to build a new, almost magical inter-region replication mode that I am very excited about, which combines synchronous replication to a "satellite" datacenter within region with asynchronous replication between regions and recovery logic that will finish replication and fail over in case of a partial failure of a region. You get fast transaction commits (much less than the inter region ping time), can fail over to a secondary region automatically and safely (without losing any committed transactions) in the vast majority of circumstances, and in the worst case you can (manually, because you are accepting data loss!) give up very recently committed transactions to fail over.

Re: Apple open-sources FoundationDB

#127

Post by pritambaral »

foobaw wrote: post #24
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

It is a genius move from Apple. I just wish they'd apply this logic to a lot of their other stuff.

~~How is it different from when Apple acquired the then-open-source FoundationDB (and shut down public access)? They could have just kept it open source back then.~~

EDIT: My bad, looks like FoundationDB wasn't fully open-source back then.

Re: Apple open-sources FoundationDB

#128

Post by hangonhn »

voidmain wrote: post #72
misterbowfinger wrote: post #43

I hate to be that person, but when I hear "ACID transactions in a distributed database", I hear Citus/Spanner/CockroachDB. I'm positive that Citus & Spanner are quite different from FoundationDB, but I have no idea how. Googling didn't help much. Can someone provide an overview of the differences?

I think Citus is not really ACID. Spanner (and to an extent its less mature OSS descendants Cockroach and TiKV) has more comparable goals, but is fairly different architecturally. For example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance (even lots of databases with much weaker guarantees are in the latter category!), doesn't trust clocks at all, doesn't lose performance when…

"or example, FoundationDB only requires N+1 replicas instead of 2N+1 to achieve N failure tolerance"

Wait, don't you need 3N+1 to tolerate N number of failures for it to be Byzantine fault tolerant? Is that not a goal of FoundationDB?

Re: Apple open-sources FoundationDB

#129

Post by voidmain »

infogulch wrote: post #119
voidmain wrote: post #103

Earlier quoted context omitted.

It is serializable and totally uncompromising. Philosophically pretty much everything defaults to the safest possible thing. It can't tolerate N failures from N+1 nodes . It can tolerate N failures with N+1 copies of your data . In a big cluster you have plenty of nodes but storing everything 5 times to tolerate 2 failures is really expensive.

> It can't tolerate N failures from N+1 copies of your data Sorry I got the terminology wrong, but that's a distinction without a difference. If it can tolerate N failures from N+1 copies, that means a network partition would allow any one copy to continue chugging along making changes by itself. You have two options: consistency is dropped and you downgrade to eventually consistent (at best), or availability is drop…

FoundationDB stores 2N+1 copies of some "coordination state" and does a consensus algorithm whenever it is updated. But this state doesn't contain a copy of your data; basically think of it as storing a replication configuration. It's very small and rarely changes.

In the happy case, replication takes place using the replicas and quorum rules specified by this configuration. For example, you might require writes to succeed synchronously against all N+1 replicas of some transaction log. After N failures, there will still be 1 replica remaining with the latest transactions. But in order to proceed after any failures, you have to do a consensus transaction against a majority of replicas of the coordination state, to specify the new set of N+1 replicas you will be using. And you also make sure that the 1 replica you are recovering from knows you are doing it, so that it won't continue to accept writes under the old replication configuration.

There can't be two partitions capable of committing transactions, because (in this case) you need either

(a) All N+1 replicas of the log, so that you can commit synchronously, or (b) A majority (N+1 out of 2N+1) of the replicas of the coordination state, AND 1 replica of the log

Sorry if this isn't a great explanation. Anyway it does work. I expect that you could rephrase this as an optimization of a consensus protocol, though I think it would be hard to build a performant and realistically featureful implementation that way.

Re: Apple open-sources FoundationDB

#130

Post by mixmastamyk »

djtriptych wrote: post #51

I went to the same high school as the founders[1]. They were about the 2 best software engineers in a school with a LOT of very smart software engineers. Another pair founded Yext, which went public last year. I still consider that school the group with the highest concentration of raw brain power I've ever been a part of. I'm probably a 1% engineer, been hired by M$, FB, and Google. These guys were light years ahead…

I did chuckle at the part above about the "best software engineers in high school," but can't argue with the results.
Post reply on HN