How does it compare to CockroachDB or TiDB?
Apple open-sources FoundationDB
61–70 of 453 posts
Re: Apple open-sources FoundationDB
#62Google please take a few notes here: 1. It’s in its own repo 2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links. 3. They use a common build system and not an in-house thing.
Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build. Although it starts by running `make`, it's about as in-house as a t…
Ah, GNU make second expansion... glad to see someone else appreciates it ;-).
Re: Apple open-sources FoundationDB
#63This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…
I'm not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I'm trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.
Honest question, does MongoDB not work for this?
Re: Apple open-sources FoundationDB
#64Anyone wants to give a summary of what primitives/structures/algorithms FoundationDB uses?
Re: Apple open-sources FoundationDB
#65Earlier quoted context omitted.
An interesting fact is that Apple is probably one of the biggest users, if not the biggest user, of Cassandra out there. Can't speak to HBase, but one thing Cassandra doesn't guarantee is ACID - I've seen some data consistency issues that has arisen from Cassandra in our usage, although it hasn't been a huge problem for us. That difference alone probably brings a lot of value to FoundationDB.
My understanding of hbase (which could be wrong) is that writes of a particular key always go to the region master first, so if you read from the master you always get the latest value of a key. The tricky part is that when the region master goes offline another one needs to take its place, and you can get inconsistency or unavailability depending on how it is set up. I’d like to see a deep dive of how foundationdb h…
By default in HBase, each region is served by a single region server. Writes and reads always go to that regionserver, and are consistent across a single row (HBase is a CP system). In this mode, when a regionserver goes offline its regions are (eventually) reassigned to a different region server. During this process the region is unavailable, but there's no loss of consistency.
Hbase 0.98 (I think?) introduced a feature called "timeline consistency" that allows reads from replica regionservers. This has to be enabled for a table and has to be specified on the query side. If it is, you have the option of falling back to the replica if the primary doesn't respond within a deadline. This may be a good tradeoff if you value availability over consistency.
Re: Apple open-sources FoundationDB
#66Re: Apple open-sources FoundationDB
#67Does anyone know if the SQL layer was open sourced, or whether there are any plans to open source it?
Re: Apple open-sources FoundationDB
#68Wow this is super exciting. I had a half dozen things that I though FDB would be good for and then poof! it got sucked up into the Apple spaceship. Now to have it emerge unscathed is pretty awesome.
Another step closer to a library appliance.
Re: Apple open-sources FoundationDB
#69Do we know where/how Apple uses FoundationDB in production ?
Wavefront's co-founder just tweeted this: https://twitter.com/panghy/status/987022825457266689
Finally it's out! @WavefrontHQ managaes petabyte scale clusters with #foundationdb today!
Re: Apple open-sources FoundationDB
#70I hadn't heard of FoundationDB before, so I did some digging into the features: https://apple.github.io/foundationdb/features.html . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won't slow down operations, and that it allows true interactive queries. I didn't think an MVCC implementation could provide that level of isolation, and I'm not eve…
I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…
Thanks, that explanation is really helpful!