Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

181–190 of 453 posts

Re: Apple open-sources FoundationDB

#181

Post by maccam94 »

jtchang wrote: post #131
wwilson wrote: post #98

Earlier quoted context omitted.

A subset of the processes in a FoundationDB cluster have the job of maintaining coordination state (via disk Paxos). In any partition situation, if one of the partitions contains a majority of the coordinators then it will stay live, while minority partitions become unavailable.

What if the number of boxes are even? Or do you get around this by not deploying an uneven number of boxes?

The majority is always N/2 + 1, where N is the number of members. A 6 member is less fault-tolerant than a 5 member cluster (quorum is 4 nodes instead of 3, and it still only allows for 2 nodes to fail).

Re: Apple open-sources FoundationDB

#182

Post by panghy »

panghy wrote: post #161

We (Wavefront) has been operating petabyte scale clusters for the last 5 years with FoundationDB (we got the source code via escrow) and we are super excited to be involved in the opensourcing of FDB. We have operated over 50 clusters on all kinds of aws instances and I can talk about all the amazing things we have done with it. https://www.wavefront.com/wavefront-foundationdb-open-source...

We basically replaced mySQL, Zookeeper and HBase with a single KV store that supports transactions, watches, and scales. It's not a trivial point that you can just develop code against a single API (finally Java 8 CompletableFutures) and not have to set up a ton of dependencies when you are building on top of FDB. We are (obviously) experts at monitoring FoundationDB with Wavefront and we hope to release the metric harvesting libraries and template dashboards that we use to do so.

Almost 5 years in and we have not lost any data (but we have lost machines, connectivity, seen kernel panics, EBS failures, SSD failures, etc., your usual day in AWS =p).

Re: Apple open-sources FoundationDB

#183

Post by wll »

voidmain wrote: post #174
wll wrote: post #154

Earlier quoted context omitted.

How would FoundationDB stay externally consistent with asynchronous cross-region replication? Thank you for your time and FoundationDB—along with @nlavezzo, and team(s)!

The satellite mode that I described is an active/passive mode. One region is accepting reads and writes; the other is just replicating everything. When it looks like the active region is in trouble, the asynchronous replication is "finished up" before switching over to the other region. The multiple datacenters in each region ensure that usually a regional failure will be "slow enough" that this automatic process (wh…

While that’s perfect to shield from orbital laser cannons, is active/active geo-independent replication possibile?

Re: Apple open-sources FoundationDB

#184

Post by voidmain »

jbit84 wrote: post #176
voidmain wrote: post #46

Earlier quoted context omitted.

I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…

Thanks. Can you elaborate on how 6 is actually accomplished? Various earlier comments have hinted that the transactional authority (conflict checking) can actually scale 'horizontally' beyond the check-throughput that can be archived by a single node. Is that the case? and whats the magic sauce for doing that for multi-object transactions? :)

Yes, conflict resolution is for most workloads a pretty small fraction of total resource use so you usually don't need a ton of resolvers (I think out of the box it still comes configured with just one?), but it can scale conflict resolution horizontally.

The basic approach isn't super hard to understand, though the details are tricky. The resolvers partition the keyspace; a write ordering is imposed on transactions and then the conflict ranges of each transaction are divided among the resolvers; each resolver returns whether each transaction conflicts and transactions are aborted if there are any conflicts.

(In general the resolution is sound, but not exact - it is possible for a transaction C to be aborted because it conflicts with another transaction B, but transaction B is also aborted because it conflicts with A (on another resolver), so C "could have" been committed. When Alec Grieser was an intern at FoundationDB he did some simulations showing that in horrible worst cases this inaccuracy could significantly hurt performance. But in practice I don't think there have been a lot of complaints about it.)

Re: Apple open-sources FoundationDB

#185

Post by evanweaver »

spullara wrote: post #173
evanweaver wrote: post #164

Earlier quoted context omitted.

It's closest to TiDB's key-value layer; a building block for more complex systems. More traditional, monolithic databases like CockroachDB (SQL) or FaunaDB (NoSQL) trade off extensibility for the benefits in performance and operations that come from very tight coupling. In my understanding, FoundationDB's transaction management is closest to FaunaDB's; read/write sets are linearized in memory in preprocessing nodes a…

I don't recall them ever having separate processes by default. You can set that up though by limiting roles.

Maybe it was the same binary. Pretty sure you had to operationally manage the roles separately and now they are integrated.

On the other hand, I don't really know.

Re: Apple open-sources FoundationDB

#186

Post by tinco »

wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Would it be possible to build a tree DB on top of it like MonetDB/Xquery? I always wondered why XML databases never took off, I've never seen anything else quit as powerful. Document databases if du jour seem comparatively lame.

Re: Apple open-sources FoundationDB

#187

Post by voidmain »

wll wrote: post #183
voidmain wrote: post #174

Earlier quoted context omitted.

The satellite mode that I described is an active/passive mode. One region is accepting reads and writes; the other is just replicating everything. When it looks like the active region is in trouble, the asynchronous replication is "finished up" before switching over to the other region. The multiple datacenters in each region ensure that usually a regional failure will be "slow enough" that this automatic process (wh…

While that’s perfect to shield from orbital laser cannons, is active/active geo-independent replication possibile?

Well, if you want ACID then you are going to have to pay for at least one geographic round trip per committed transaction. (So why not go active/passive, and have at least one of your datacenters be fast?)

But what if you have different pieces of data and you want them to be fast in different datacenters? I think a great solution to this can be layered on top of multiple FoundationDB clusters, each using the satellite mode, but this is one thing that I at least haven't been able to think of a way to provide properly at the data model agnostic key/value store layer - the details about what to put where seem fundamentally dependent on your data model.

Re: Apple open-sources FoundationDB

#188

Post by verulito »

djtriptych wrote: post #51

I went to the same high school as the founders[1]. They were about the 2 best software engineers in a school with a LOT of very smart software engineers. Another pair founded Yext, which went public last year. I still consider that school the group with the highest concentration of raw brain power I've ever been a part of. I'm probably a 1% engineer, been hired by M$, FB, and Google. These guys were light years ahead…

Just checked the rankings. My first high school made #172 on the list ("International School of the Americas").

My second high school is unranked ("Texas Academy of Math and Science"). I don't think it qualifies as a high school. Seems we haven't done a great job of identifying the accomplishments of our alumni, based on the Wikipedia page. No doubt it would rank near the top though. My year alone Caltech accepted about 30 of us, more than any other high school in the country. Makes me wonder what my peers have been up to.

Anyway, I'd agree that these tech high schools have some amazingly smart people attending them.

2: https://en.wikipedia.org/wiki/Texas_Academy_of_Mathematics_a...

Re: Apple open-sources FoundationDB

#189

Post by aseipp »

wwilson wrote: post #76
itp wrote: post #55

Earlier quoted context omitted.

As an end user it's absolutely better. I'm just torn between pride and embarrassment thinking about how it was implemented.

I think my favorite part of the build process was when we frobnicated libstdc++: https://github.com/apple/foundationdb/blob/master/Makefile#L... And then there were the hijinks we went through to build a cross-compiler with modern gcc and ancient libc (plus the steps to make sure no dependency on later glibc symbols snuck in): https://github.com/apple/foundationdb/blob/master/build/link... Ahh... now that was a build…

I saw this in the Makefiles, and -- ah, the life of distributing proprietary Linux software. At one point in a prior job, we just replaced our build system with a wrapper Makefile that 'chroot'ed into a filesystem image that was a snapshot of one of our build machines, since it was so difficult to set up. This meant we (developers) had easier system updates, security upgrades, etc. That was just the tip of the iceberg!

Re: Apple open-sources FoundationDB

#190

Post by jayrhynas »

zzzcpan wrote: post #50
throwaway6497 wrote: post #44

Broken documentation: Architecture image is missing. https://apple.github.io/foundationdb/architecture.html Hope someone from the team reads this.

It's a pdf: https://apple.github.io/foundationdb/_images/Architecture.pd...

Looks like Safari will happily load a pdf in an img tag, but Chrome won't
Post reply on HN