Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

211–220 of 453 posts

Re: Apple open-sources FoundationDB

#211

Post by fizx »

voidmain wrote: post #194
bryanrasmussen wrote: post #171

Earlier quoted context omitted.

How would you replace a Lucene/Elasticsearch index with foundationDb?

It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.

It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely.

If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well.

Doing a Lucene+DB implementation where each entry posting lists are stored natively in the key-value system was explored for Lucene+Cassandra as (https://github.com/tjake/Solandra). It was horrifically slow, not because Cassandra was slow, but because posting lists are optimized and putting them in a generalized b-tree or LSM-tree variant will remove some locality and many of the possible optimizations.

I'm still holding out some hope for a hybrid implementation where posting list ranges are stored in a kv store.

Re: Apple open-sources FoundationDB

#212

Post by voidmain »

wll wrote: post #202
voidmain wrote: post #187

Earlier quoted context omitted.

Well, if you want ACID then you are going to have to pay for at least one geographic round trip per committed transaction. (So why not go active/passive, and have at least one of your datacenters be fast?) But what if you have different pieces of data and you want them to be fast in different datacenters? I think a great solution to this can be layered on top of multiple FoundationDB clusters, each using the satellit…

> So why not go active/passive, and have at least one of your datacenters be fast? While local writes would stay fast, wouldn’t active/passive see higher-latency non-local writes than Spanner or Fauna’s (assuming a NAM-EUR-ASIA topology)? I agree with and do appreciate the multiple FoundationDB clusters suggestion.

I'm speculating, but I think in this mode, from the "slow" datacenters you would see one round trip time to start a transaction, then reads will be fast (they can be done safely from your local datacenter because of MVCC), and then one round trip time to commit the transaction. I think that's as good as Spanner does with the same geography, but I'm not sure. I think you could get rid of the first round trip time even without any clock synchronization nonsense, by speculating on a read version for read/write transactions. And 1xRTT is obviously as fast as physically possible for ACID.

Re: Apple open-sources FoundationDB

#213

Post by rcgenova »

voidmain wrote: post #4
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Will said what I wanted to say, but: me too. I'm super happy about this and grateful to the team that made it happen! (I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it's open source, I can rejoin the community!)

Ditto. So glad to see FoundationDB available again! Both the tech and the company have been missed. (Former FoundationDB Solutions Engineer :) )

Re: Apple open-sources FoundationDB

#214

Post by ccleve »

voidmain wrote: post #129
infogulch wrote: post #119

Earlier quoted context omitted.

> It can't tolerate N failures from N+1 copies of your data Sorry I got the terminology wrong, but that's a distinction without a difference. If it can tolerate N failures from N+1 copies, that means a network partition would allow any one copy to continue chugging along making changes by itself. You have two options: consistency is dropped and you downgrade to eventually consistent (at best), or availability is drop…

FoundationDB stores 2N+1 copies of some "coordination state" and does a consensus algorithm whenever it is updated. But this state doesn't contain a copy of your data; basically think of it as storing a replication configuration. It's very small and rarely changes. In the happy case, replication takes place using the replicas and quorum rules specified by this configuration. For example, you might require writes to s…

Yup, this works. I wonder if this is the first commercial application of this insight? http://hh360.user.srcf.net/blog/2016/08/majority-agreement-i...

Can you tell us what consensus algorithm you're using? Raft, or something else? Your own implementation or something off the shelf?

Re: Apple open-sources FoundationDB

#215

Post by shawn-butler »

jasode wrote: post #114
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

>It’s in widespread production use at Apple Anybody know if Apple migrated any projects from Cassandra to FoundationDB? Or was every project using FoundationDB a greenfield project?

There were rumors iMessage moved there but ¯\_(ツ)_/¯

Would be great to see from Apple an engineering blog "strengths and challenges at scale" post now that it has been opened again.

Re: Apple open-sources FoundationDB

#216

Post by ddorian43 »

fizx wrote: post #211
voidmain wrote: post #194

Earlier quoted context omitted.

It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.

It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely. If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well. Doing a Lucene+DB impleme…

See elassandra for a better solandra, keeping lucene indexes and sstables separate. It should be better than keeping posting list in kv-store

Re: Apple open-sources FoundationDB

#217

Post by spullara »

tinco wrote: post #186
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Would it be possible to build a tree DB on top of it like MonetDB/Xquery? I always wondered why XML databases never took off, I've never seen anything else quit as powerful. Document databases if du jour seem comparatively lame.

They had a document layer that wasn't released that was MongoDB compatible — so you can do this.

Re: Apple open-sources FoundationDB

#218

Post by 15155 »

ccifuentes wrote: post #136
wwilson wrote: post #2

This is INCREDIBLE news! FoundationDB is the greatest piece of software I’ve ever worked on or used, and an amazing primitive for anybody who’s building distributed systems. The short version is that FDB is a massively scalable and fast transactional distributed database with some of the best testing and fault-tolerance on earth[1]. It’s in widespread production use at Apple and several other major companies. But the…

Do we really need another rookie database system? To be honest, Postgres and MongoDB is all you need to achieve any project. I respect hobby projects though, and if that’s the case then great!

> I respect hobby projects though

https://techcrunch.com/2015/03/24/apple-acquires-durable-dat...

FDB raised >$22m and was acquired by Apple.

If you consider that a "hobby project," please, teach me how to hobby.

Re: Apple open-sources FoundationDB

#219

Post by 15155 »

shawn-butler wrote: post #205
SubuSS wrote: post #110

This is great news, when I was with dynamo, FoundationDB was the other green shore for me :). They did so many things so well. A tiny bit of caution for folks trying to run systems like this though: It is frigging hard at any reasonable scale. The whole thing might be documented / OSS and what not, but very soon you are going to run into deep enough problems that's going to require very core knowledge to debug, energ…

This sounds like a business opportunity to me rather than a cautionary note. I remember all the people who bashed Apple when they acquired FoundationDB. I hope they are appropriately ashamed now.

> I remember all the people who bashed Apple when they acquired FoundationDB.

I'm not ashamed about deriding apple for Apple taking a really, really great product and hiding it from the world for years to come.

This is definitely some atonement, but does not totally absolve Apple from the many times they've taken tech private.

Re: Apple open-sources FoundationDB

#220

Post by voidmain »

infogulch wrote: post #191
voidmain wrote: post #177

Earlier quoted context omitted.

You only have to write to the coordination state when there is a failure. You can commit millions of transactions in the happy case without ever doing such a write. And failure detector performance and other engineering concerns are usually more of a limitation, in practice, on the performance of recovery than the latency of the coordination state consensus, even when the coordinators are geographically distributed.

So the strategy is to optimistically assume that there are no failures and just replicate to all N+1 copies. If there's a failure then back off to the consensus state to coordinate the fix rigorously. In the best case with no failures this works great. But as the number of failures increases, I feel like due to the extra synchronization there will be an inflection point where the cost of the extra layers of coordinat…

If you expect to have lots of (hopefully very temporary!) node failures, I think FoundationDB has another trick up its sleeve. You can store (say) N+2 replicas of transaction logs, which are also relatively small and (since sequential) efficient. Then you have a write quorum of N+1 and a recovery quorum of 2 logs, and you don't have to do coordination on every failure.

It's certainly true that with enough failures you aren't going to make much progress. I'm not sure that is any less true with plain old state machine replication protocols, though.

Post reply on HN