Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

41–50 of 453 posts

Re: Apple open-sources FoundationDB

#41

Post by poooogles »

Joeri wrote: post #38
Bahamut wrote: post #18

Earlier quoted context omitted.

An interesting fact is that Apple is probably one of the biggest users, if not the biggest user, of Cassandra out there. Can't speak to HBase, but one thing Cassandra doesn't guarantee is ACID - I've seen some data consistency issues that has arisen from Cassandra in our usage, although it hasn't been a huge problem for us. That difference alone probably brings a lot of value to FoundationDB.

My understanding of hbase (which could be wrong) is that writes of a particular key always go to the region master first, so if you read from the master you always get the latest value of a key. The tricky part is that when the region master goes offline another one needs to take its place, and you can get inconsistency or unavailability depending on how it is set up. I’d like to see a deep dive of how foundationdb h…

From what I understand that shouldn't happen as writes are written to a WAL on HDFS (been a while since I've used it).

From my memory writes within a row are atomic. It seems to pass Jepsen as well [1].

1. https://www.google.co.uk/amp/s/yokota.blog/2015/09/30/call-m...

Re: Apple open-sources FoundationDB

#42

Post by jgh »

itp wrote: post #40
_xzxj wrote: post #32

Google please take a few notes here: 1. It’s in its own repo 2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links. 3. They use a common build system and not an in-house thing.

Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build. Although it starts by running `make`, it's about as in-house as a t…

Fair enough, I stand corrected. Superficially it seems like a more pleasant experience than dealing with gn/ninja as an end user.

Re: Apple open-sources FoundationDB

#43

Post by misterbowfinger »

I hate to be that person, but when I hear "ACID transactions in a distributed database", I hear Citus/Spanner/CockroachDB.

I'm positive that Citus & Spanner are quite different from FoundationDB, but I have no idea how. Googling didn't help much.

Can someone provide an overview of the differences?

Re: Apple open-sources FoundationDB

#46

Post by voidmain »

openasocket wrote: post #10

I hadn't heard of FoundationDB before, so I did some digging into the features: https://apple.github.io/foundationdb/features.html . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won't slow down operations, and that it allows true interactive queries. I didn't think an MVCC implementation could provide that level of isolation, and I'm not eve…

I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency.

A FDB transaction roughly works like this, from the client's perspective:

1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction

2. Do reads from a consistent MVCC snapshot at that read version. No matter what other activity is happening you see an unchanging snapshot of the database. Keep track of what (ranges of) data you have read

3. Keep track of the writes you would like to do locally.

4. If you read something that you have written in the same transaction, use the write to satisfy the read, providing the illusion of ordering within the transaction

5. When and if you decide to commit the transaction, send the read version, a list of ranges read and writes that you would like to do to the distributed database.

6. The distributed database assigns a write version to the transaction and determines if, between the read and write versions, any other transaction wrote anything that this transaction read. If so there is a conflict and this transaction is aborted (the writes are simply not performed). If not then all the writes happen atomically.

7. When the transaction is sufficiently durable the database tells the client and the client can consider the transaction committed (from an external consistency standpoint)

The implementations of 1 and 6 are not trivial, of course :-)

So a sufficiently "slow client" doing a read write transaction in a database with lots of contention might wind up retrying its own transaction indefinitely, but it can't stop other readers or writers from making progress.

It's still the case that if you want great performance overall you want to minimize conflicts between transactions!

Re: Apple open-sources FoundationDB

#47

Post by cnlwsu »

protomyth wrote: post #33

Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don't provide a distribution that will work on it. I know the license says Apple doesn't have to do anything, but it just seems wrong that they didn't provide a download.

https://www.foundationdb.org/download/

Re: Apple open-sources FoundationDB

#48

Post by jaboutboul »

protomyth wrote: post #33

Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don't provide a distribution that will work on it. I know the license says Apple doesn't have to do anything, but it just seems wrong that they didn't provide a download.

I’m sure one will be shortly forthcoming. They did a good job up actually putting up the code quickly following (in parallel) with the announcement.

Re: Apple open-sources FoundationDB

#49

Post by jaboutboul »

cnlwsu wrote: post #47
protomyth wrote: post #33

Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don't provide a distribution that will work on it. I know the license says Apple doesn't have to do anything, but it just seems wrong that they didn't provide a download.

https://www.foundationdb.org/download/

And there we go...
Post reply on HN