I wish Apple just became a customer instead of buying and burying this for 3 years. Glad to see it come back but community momentum is hard to rebuild.
Apple open-sources FoundationDB
301–310 of 453 posts
Re: Apple open-sources FoundationDB
#302Earlier quoted context omitted.
We basically replaced mySQL, Zookeeper and HBase with a single KV store that supports transactions, watches, and scales. It's not a trivial point that you can just develop code against a single API (finally Java 8 CompletableFutures) and not have to set up a ton of dependencies when you are building on top of FDB. We are (obviously) experts at monitoring FoundationDB with Wavefront and we hope to release the metric h…
"but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS " <=== This I wish more people realized that is a day to day reality if you are in AWS at scale.
As I understand it, it's like that everywhere at scale, not just on AWS, it being a property of operating at scale.
Or are you saying that AWS is particularly unreliable at scale?
Re: Apple open-sources FoundationDB
#303Earlier quoted context omitted.
Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.
Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project
I believe the "d" in the GP comment was not a typo https://news.ycombinator.com/item?id=16874734
Re: Apple open-sources FoundationDB
#304Earlier quoted context omitted.
This is a good explanation of how it happens on a single node. What do you do when the transaction is distributed? How do you achieve consensus? Is there a write up on it anywhere?
The only thing that's different in a distributed cluster is the implementations of steps 1 and 6. As voidmain said, the details of that are not trivial, ESPECIALLY the details of how it never produces wrong answers during fault conditions. I don't know that there's been an exhaustive writeup of that part, but maybe one of us or somebody on the Apple team will put something together. It probably won't fit in an HN com…
The documentation ( https://apple.github.io/foundationdb/technical-overview.html ) sells the product, but doesn't give a deep enough explanation. As a closed source product, that's understandable.
Going forward as an opensource product, I hope to see some clarity on the "how it works"... Distributed, performant ACID sounds good, almost too good to be true. Not that I doubt it at the moment, I just want to understand it better :)
Re: Apple open-sources FoundationDB
#305Earlier quoted context omitted.
Thanks for the detailed answer. Is it actually serializable isolation - does it handle write skew anomalies ( https://en.wikipedia.org/wiki/Snapshot_isolation )? Most OCC systems I know have only snapshot isolation. Systems that sound closest to FoundationDB's transaction model that i can think of are Omid ( https://omid.incubator.apache.org/ ) and Phoenix ( https://phoenix.apache.org/transactions.html ). They both s…
I agree with voidmain’s comment as secondary indexes shouldn’t be any different than the primary KV in your case. Almost seems that you’re focusing on a SQL/Relational database architecture but storing your data demoralized anyways. Odd combination of thoughts.
I love the idea of demoralized data :)
Re: Apple open-sources FoundationDB
#306Earlier quoted context omitted.
It has the ability to watch keys so building a notification system on top of it is pretty easy. Really only limited by your imagination.
In addition to single-key asynchronous watches, there are also versionstamped ops (for maintaining your own, sophisticated log in a layer) and configurable key range transaction logging (but see the caveats in my other post on the topic). I'm not sure it has every feature it will ever need in this area, but it's a pretty good starting point for building "reactive" stuff.
Your response is both very exciting and slightly intimidating! Would love to see a “NewSQL” and/or event store built on top of FDB tech. Would the key ranges and versioned ops be capable of providing/emulating a performant “atomic broadcast” similar to that in Kafka?
Re: Apple open-sources FoundationDB
#307Earlier quoted context omitted.
I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…
Does the implementation handle the case that you want to do a write that is conditioned on a prior read finding no corresponding record(s)?
Of course. FDB thinks about read and write conflict ranges, which are functions of the keys, not the values (or lack thereof). A read of a non-existent key conflicts with a write to that key. A read of a range of keys conflicts with a write to a key in that range, even if that key did not have an associated value at the time of the original read.
Re: Apple open-sources FoundationDB
#308This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…
> How about replacing your Lucene/ElasticSearch index with something that actually scales and works?
Do you have something to back that up? This to me reads like you imply that Elasticsearch does not work and scale.
It's definitely interesting but I'm cautious. The track record for FoundationDB and Apple has not been great here. IIRC they acquired the company and took the software offline leaving people in the rain?
Could this be like it happened with Cassandra at Facebook where they dropped the code and then more or less stopped contributing?
Also I haven't seen many contributions from Apple to open-source projects like Hadoop etc. in the past few years. Looking for "@apple.com" email addresses in the mailing lists doesn't yield a lot of results. I understand that this is a different team and that people might use different E-Mail addresses and so on.
In general I'm also always cautious (but open-minded) when there's lots of enthusiasm and there seems to be no downside. I'm sure FoundationDB has its dirty little secrets and it would be great to know what those are.
Re: Apple open-sources FoundationDB
#309Earlier quoted context omitted.
Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.
Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project
Clangd != clang.
Woolvalley said “and now recently clangd“, which is a source code completion server which began its life at Google.
Clang did start its life with Apple.
Re: Apple open-sources FoundationDB
#310Earlier quoted context omitted.
Apple also uses HBase for Siri I believe, what are some of the cluster sizes that FoundationDB scales to? Could it be used to replace HBase or Hadoop? I was in attendance at your talk, and thought it was one of the best at the conference. Apple I think broke some hearts completely going closed-source for a while, but glad to see them open sourcing a promising technology.
If scale is a function of read/writes, very large. In fact with relatively minimal (virtual) hardware it's not insane to see a cluster doing around 1M writes/second.
I was talking more about large file storage like HDFS, and the MapReduce model of bringing computation to data. HBase does the latter, and it's strongly consistent like FoundationDB, though FoundationDB provides better guarantees. As a K/V I understand what you and OP say.