FoundationDB Record Layer

#1

Post by davelester » Mon, Jan 14, 2019, 8:51 PM UTC

FoundationDB Record Layer
foundationdb.org

Re: FoundationDB Record Layer

#2

Post by wwilson » Mon, Jan 14, 2019, 8:53 PM UTC

This is very cool!

FoundationDB excites a lot of people because it's an extremely scalable and extremely reliable distributed database that supports ACID transactions, and which is both open-source and has Apple standing behind it. And yeah, all of that is pretty nice.

But arguably the real power comes from the fact that it exposes a relatively low-level data model that can then be wrapped in one or more stateless "layers". All of these layers write to the same storage substrate, so you can have your document database, your SQL database, your time-series database, your consensus/coordination store, your distributed task queue, etc., etc., but you're only actually operating one stateful system. Your SREs will thank you.

Writing these layers to be scalable and high-performance can be challenging, but it looks like Apple is actively doing it and willing to release the results to the rest of us. This also suggests that their previous open-sourcing of the MongoDB-compatible document layer wasn't a one-off fluke. All of this is very good news for everybody who needs to run databases in the real world.

Full disclosure: I worked on FoundationDB a long, long time ago.

Re: FoundationDB Record Layer

#3

Post by wwilson » Mon, Jan 14, 2019, 8:59 PM UTC

wwilson wrote: post #2

This is very cool! FoundationDB excites a lot of people because it's an extremely scalable and extremely reliable distributed database that supports ACID transactions, and which is both open-source and has Apple standing behind it. And yeah, all of that is pretty nice. But arguably the real power comes from the fact that it exposes a relatively low-level data model that can then be wrapped in one or more stateless "l…

Wow, from the post:

"Together, the Record Layer and FoundationDB form the backbone of Apple's CloudKit. We wrote a paper describing how we built the Record Layer to run at massive scale and how CloudKit uses it."

I think this is the first time that little detail has been publicly disclosed.

Re: FoundationDB Record Layer

#4

Post by ryanworl » Mon, Jan 14, 2019, 9:27 PM UTC

Congrats to the team at Apple for getting this released! They have had a busy few months with getting the document layer released, the FDB Summit, and now the record layer.

Re: FoundationDB Record Layer

#5

Post by gigatexal » Mon, Jan 14, 2019, 9:28 PM UTC

This is super exciting. Can’t wait to have some time this weekend to play with it.

Re: FoundationDB Record Layer

#6

Post by manigandham » Mon, Jan 14, 2019, 9:29 PM UTC

Glad Apple is releasing all of this, I wonder what kickstarted it all?

The paper is rather interesting: https://www.foundationdb.org/files/record-layer-paper.pdf

Re: FoundationDB Record Layer

#7

Post by jwr » Mon, Jan 14, 2019, 9:41 PM UTC

Very interesting. I've been looking closely at FoundationDB as a way forward (to replace RethinkDB and Cassandra in existing systems). It's one of the few contenders for a really interesting take on a distributed database.

I am not sure if I will use the record layer (I've been planning to write "my layer" myself), but it will definitely be an interesting thing to look at.

Re: FoundationDB Record Layer

#8

Post by rubyn00bie » Mon, Jan 14, 2019, 9:46 PM UTC

wwilson wrote: post #2

This is very cool! FoundationDB excites a lot of people because it's an extremely scalable and extremely reliable distributed database that supports ACID transactions, and which is both open-source and has Apple standing behind it. And yeah, all of that is pretty nice. But arguably the real power comes from the fact that it exposes a relatively low-level data model that can then be wrapped in one or more stateless "l…

Just wanted to say thanks for your testing distributed systems talk! That has been a huge help to me as I develop distributed systems and consider the problems inherent to them.

Re: FoundationDB Record Layer

#9

Post by _y4bi » Mon, Jan 14, 2019, 9:56 PM UTC

jwr wrote: post #7

Very interesting. I've been looking closely at FoundationDB as a way forward (to replace RethinkDB and Cassandra in existing systems). It's one of the few contenders for a really interesting take on a distributed database. I am not sure if I will use the record layer (I've been planning to write "my layer" myself), but it will definitely be an interesting thing to look at.

Fellow RethinkDB user here. I’ve been looking at Cassandra and FoundationDB as replacements. I’m genuinely curious— what didn’t you like about Cassandra?

Re: FoundationDB Record Layer

#10

Post by nathan_long » Mon, Jan 14, 2019, 10:07 PM UTC

wwilson wrote: post #2

This is very cool! FoundationDB excites a lot of people because it's an extremely scalable and extremely reliable distributed database that supports ACID transactions, and which is both open-source and has Apple standing behind it. And yeah, all of that is pretty nice. But arguably the real power comes from the fact that it exposes a relatively low-level data model that can then be wrapped in one or more stateless "l…

This definitely sounds cool.

> your SQL database

They mention "a declarative query API", but as far as I can tell that's not actually SQL, right? So migrating from another relational db would require learning a new query language?