Apple open-sources FoundationDB • Hacker News Board

Re: Apple open-sources FoundationDB

#31

Post by spullara » Thu, Apr 19, 2018, 5:20 PM UTC

arthursilva wrote: post #30

This is great news! Unfortunately it looks like they striped out some important things, notably the storage engine (there's now a sqlite fallback). Edit: Apparently it was always sqlite as per replies bellow.

The storage engine has always been SQLite for the SSD engine.

Re: Apple open-sources FoundationDB

#32

Post by _xzxj » Thu, Apr 19, 2018, 5:20 PM UTC

Google please take a few notes here:

1. It’s in its own repo

2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links.

3. They use a common build system and not an in-house thing.

Re: Apple open-sources FoundationDB

#33

Post by protomyth » Thu, Apr 19, 2018, 5:21 PM UTC

Apple builds their OS with a lot of software from FreeBSD, but when they opensource FoundationDB they don't provide a distribution that will work on it. I know the license says Apple doesn't have to do anything, but it just seems wrong that they didn't provide a download.

Re: Apple open-sources FoundationDB

#34

Post by nlavezzo » Thu, Apr 19, 2018, 5:21 PM UTC

voidmain wrote: post #4

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Will said what I wanted to say, but: me too. I'm super happy about this and grateful to the team that made it happen! (I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it's open source, I can rejoin the community!)

Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it's fantastic that it's available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.

Re: Apple open-sources FoundationDB

#35

Post by Osmium » Thu, Apr 19, 2018, 5:22 PM UTC

Original thread for when Apple acquired FoundationDB for those curious: https://news.ycombinator.com/item?id=9259986

Re: Apple open-sources FoundationDB

#36

Post by optimuspaul » Thu, Apr 19, 2018, 5:23 PM UTC

kawera wrote: post #14

Do we know where/how Apple uses FoundationDB in production ?

I had always assumed they acquired the team more than the ip. I'm not sure this confirms that or not. I'd be curious what the answer to this question is as well, but wouldn't be surprised if the answer was "they don't"

Re: Apple open-sources FoundationDB

#37

Post by fooster » Thu, Apr 19, 2018, 5:24 PM UTC

arthursilva wrote: post #30

This is great news! Unfortunately it looks like they striped out some important things, notably the storage engine (there's now a sqlite fallback). Edit: Apparently it was always sqlite as per replies bellow.

I don't think that is correct. My understanding from using fdb for years in production was that sqlite was the storage engine.

Re: Apple open-sources FoundationDB

#38

Post by Joeri » Thu, Apr 19, 2018, 5:24 PM UTC

Bahamut wrote: post #18

samspenc wrote: post #9

How does this stack up against HBase and Cassandra, which seem to have gotten traction already in the same areas that FoundationDB seems best suited for?

An interesting fact is that Apple is probably one of the biggest users, if not the biggest user, of Cassandra out there. Can't speak to HBase, but one thing Cassandra doesn't guarantee is ACID - I've seen some data consistency issues that has arisen from Cassandra in our usage, although it hasn't been a huge problem for us. That difference alone probably brings a lot of value to FoundationDB.

My understanding of hbase (which could be wrong) is that writes of a particular key always go to the region master first, so if you read from the master you always get the latest value of a key. The tricky part is that when the region master goes offline another one needs to take its place, and you can get inconsistency or unavailability depending on how it is set up.

I’d like to see a deep dive of how foundationdb handles this. It has to trade-off consistency for availability at some point and it would be nice to know exactly where.

Re: Apple open-sources FoundationDB

#39

Post by ifcaselet » Thu, Apr 19, 2018, 5:24 PM UTC

endlessvoid94 wrote: post #26

kawera wrote: post #14

Do we know where/how Apple uses FoundationDB in production ?

My first thought was...iCloud?

iCloud is an umbrella term for several different systems.

Re: Apple open-sources FoundationDB

#40

Post by itp » Thu, Apr 19, 2018, 5:25 PM UTC

_xzxj wrote: post #32

Google please take a few notes here: 1. It’s in its own repo 2. The build instructions are concise and clear. Dependencies are listed. You have to follow a total of 0 links. 3. They use a common build system and not an in-house thing.

Speaking as the original author of this monstrosity of a build system, please be careful before offering praise here. To be clear, there is a top-level, non-recursive Makefile that uses the second expansion feature of GNU make, translating Visual Studio project files into generated Makefile inputs that are transformed into targets to power the build.

Although it starts by running `make`, it's about as in-house as a thing can be.