Apple open-sources FoundationDB

Post by voidmain » Fri, Apr 20, 2018, 1:01 AM UTC

ddorian43 wrote: post #258

voidmain wrote: post #226

Earlier quoted context omitted.

In some distributed databases the client just connects to some machine in the cluster and tells it what it wants to do. You pay the extra latency as it redirects these requests where they should go. In FDB's envisioned architecture, the "client" is usually a (stateless, higher layer) database node itself! So the client encompasses the first layer of distributed database technology, connects directly to services throu…

Wouldn't layers be hard to be built on the server (since you have to also change the client) and slow to be built as a layer (since it will be another separate service) ?

Post by voidmain » Fri, Apr 20, 2018, 1:05 AM UTC

spullara wrote: post #247

zackmorris wrote: post #243

Looks promising, but does anyone know if FoundationDB has external events or triggers, similar to Firebase or RethinkDB? I can't seem to find much on it. If not, then a lot of potential is being left on the table, because usage would require wrapping FoundationDB in a proxy or middleware of some kind to synthesize events, which can be extremely difficult to get right (due to race conditions, atomicity issues, etc). W…

It has the ability to watch keys so building a notification system on top of it is pretty easy. Really only limited by your imagination.

Post by mzeier » Fri, Apr 20, 2018, 1:06 AM UTC

KyleBrandt wrote: post #230

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Know of any ideas around using this for long term Time Series data? Wonder if maybe something like OpenTSDB but with this as backend instead of hbase (which can be a sort of operational hell)

Post by AlphaSite » Fri, Apr 20, 2018, 1:16 AM UTC

ccleve wrote: post #214

voidmain wrote: post #129

Earlier quoted context omitted.

FoundationDB stores 2N+1 copies of some "coordination state" and does a consensus algorithm whenever it is updated. But this state doesn't contain a copy of your data; basically think of it as storing a replication configuration. It's very small and rarely changes. In the happy case, replication takes place using the replicas and quorum rules specified by this configuration. For example, you might require writes to s…

Yup, this works. I wonder if this is the first commercial application of this insight? http://hh360.user.srcf.net/blog/2016/08/majority-agreement-i... Can you tell us what consensus algorithm you're using? Raft, or something else? Your own implementation or something off the shelf?

Post by wwilson » Fri, Apr 20, 2018, 1:32 AM UTC

ccleve wrote: post #197

voidmain wrote: post #46

Earlier quoted context omitted.

I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…

This is a good explanation of how it happens on a single node. What do you do when the transaction is distributed? How do you achieve consensus? Is there a write up on it anywhere?