Apple open-sources FoundationDB • Hacker News Board

Re: Apple open-sources FoundationDB

#91

Post by emersonrsantos » Thu, Apr 19, 2018, 6:13 PM UTC

adolph wrote: post #23

"because it is an ordered key-value store, FoundationDB can use range reads to efficiently scan large swaths of data" https://apple.github.io/foundationdb/features.html I wonder how it compares to MUMPS databases like Intersystems Cache and FIS GtM?

Caché is more oriented towards vertical scaling, big monolithic servers, where FoundationDB is more towards horizontal scaling.

Also, the obvious difference is that Caché is closed source and prohibitively expensive.

FiS is only FOSS on selected platforms (Linux x86, OpenVMS Alpha), and proprietary on all other platforms.

Re: Apple open-sources FoundationDB

#92

Post by nlavezzo » Thu, Apr 19, 2018, 6:13 PM UTC

sunw wrote: post #86

nlavezzo wrote: post #85

Earlier quoted context omitted.

The core product is a distributed, highly fault tolerant ordered Key Value store with true serializable ACID transactions. All of the layers (including document, graph) sit on top of that and inherit its ACID properties, scalability, fault tolerance, etc. It doesn't appear to me that they released any of the top level layers, but those are MUCH simpler to build, and that's where the OS community can step in.

How is FoundationDB's graph performance/feature set compared to those of other graph and multi-model databases?

FoundationDB at its core is not a graph database. You could build a graph database on top of it, using FoundationDB as a very strong and feature rich storage engine, however you'd like. It would be much simpler to do than building a new (especially a distributed) graph database from scratch.

Re: Apple open-sources FoundationDB

#93

Post by jtchang » Thu, Apr 19, 2018, 6:15 PM UTC

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

I just watched the demo of 5 machines and 2 getting unplugged. The remaining 3 can form a quorum. What happens if it was 3 and 3? Do they both form quorums?

Re: Apple open-sources FoundationDB

#94

Post by nlavezzo » Thu, Apr 19, 2018, 6:15 PM UTC

kbumsik wrote: post #77

nlavezzo wrote: post #69

Earlier quoted context omitted.

Wavefront's co-founder just tweeted this: https://twitter.com/panghy/status/987022825457266689 Finally it's out! @WavefrontHQ managaes petabyte scale clusters with #foundationdb today!

I'm confused. How they could use FoundationDB when Apple aquired it a long time ago? Did apple sell the software to other companies?

FoundationDB was a company that existed in the market, licensing our database technology for quite a while. Licenses don't necessarily terminate upon acquisitions.

Re: Apple open-sources FoundationDB

#95

Post by DimitarIbra9987 » Thu, Apr 19, 2018, 6:16 PM UTC

is Transaction authority in-memory or on disk? this architecture seems kinda clunky. Wondering what's the performance and is this a good fit for quick k-v store use cases?

Re: Apple open-sources FoundationDB

#96

Post by manigandham » Thu, Apr 19, 2018, 6:16 PM UTC

xvilka wrote: post #90

These days most people know Hadoop for a distributed storage. In my opinion, though CEPH [1] has the bigger potential. [1] https://ceph.com/

There are several distributed storage systems like Ceph and they all have problems. Ceph is not good because it's an object storage system trying to provide block storage and a filesystem on top, which will never work well.

Re: Apple open-sources FoundationDB

#97

Post by tomc1985 » Thu, Apr 19, 2018, 6:18 PM UTC

I remember reading about Apple purchasing FoundationDB and everyone being disappointed and hoping they'd open source it. And now they have!

Re: Apple open-sources FoundationDB

#98

Post by wwilson » Thu, Apr 19, 2018, 6:19 PM UTC

jtchang wrote: post #93

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

I just watched the demo of 5 machines and 2 getting unplugged. The remaining 3 can form a quorum. What happens if it was 3 and 3? Do they both form quorums?

A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos).

In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.

Re: Apple open-sources FoundationDB

#99

Post by mzeier » Thu, Apr 19, 2018, 6:19 PM UTC

voidmain wrote: post #59

arthursilva wrote: post #30

This is great news! Unfortunately it looks like they striped out some important things, notably the storage engine (there's now a sqlite fallback). Edit: Apparently it was always sqlite as per replies bellow.

The storage engine is and always was a fairly heavily modified asynchronous version of sqlite's btree. It's been extremely reliable, which was always our top priority, and the performance isn't bad. But honestly when there was a problem with it our development velocity improving it wasn't great. It's super easily pluggable[1], so now that it is open source people can experiment with other engines. I think there is a…

Truthfully at Wavefront we've taken the json status directly into telegraf. Plus a bunch of python tooling to massage additional telemetry on a clusters health (coordinator reachability for example).

Plus even more tooling (mostly Ansible) for managing large fleets.

Re: Apple open-sources FoundationDB

#100

Post by voidmain » Thu, Apr 19, 2018, 6:19 PM UTC

dbcurtis wrote: post #74

Question for the FoundationDB gurus that are hanging out on this thread: How well does it deal with spotty connectivity? I'm asking because I work on mobile robots, and WiFi and/or LTE connections are always coming and going in unpredictable ways as the vehicle moves about it's environment. Reconnecting every few minutes is normal.

The fault tolerance is pretty much flawless. You won't be able to get the database "stuck" or see anomalies.

But performance is going to suck if you run server nodes over unreliable connections. I have trouble seeing a FoundationDB cluster running on mobile robots as more than a trade show gimmick. Albeit an awesome gimmick. So in summary you should totally do that.