Apple open-sources FoundationDB • Hacker News Board

Re: Apple open-sources FoundationDB

#281

Post by degenerate » Fri, Apr 20, 2018, 1:35 AM UTC

osrec wrote: post #227

I can see everyone's extremely happy about this, which is great. As someone who's never used it, I'd like to know more about FoundationDB and how it compares to other offerings such as MySQL or Postgres, and which use cases is it most suited to. I would especially love to hear the thoughts of those with direct experience of using Foundation DB. Thanks!

Seconding this. Instead of hearing from bloggers or contributors, what is using this DB like in the trenches? What simple use case would fit this DB perfectly? Is there a lot of setup on second/third deploy? Easy to maintain, or requires a lot of tweaking/tuning? How is memory usage with 10M records? 100M?

Re: Apple open-sources FoundationDB

#282

Post by bobjordan » Fri, Apr 20, 2018, 1:36 AM UTC

Nice documentation around the python API. At this point, docs state it supports 2.7-3.4. What scale of workload is required to catch it up to 3.6 compatability? https://apple.github.io/foundationdb/api-python.html

Re: Apple open-sources FoundationDB

#283

Post by josephg » Fri, Apr 20, 2018, 1:44 AM UTC

voidmain wrote: post #263

josephg wrote: post #252

This is wonderful news. I built a proof-of-concept realtime collaborative editor on top of foundationdb a few years ago, and was very disappointed when I couldn't use it in production. I'm really excited to use this in some projects I'm working on. Quick question: I know there's a watch API, but is there any way to subscribe to a change feed from foundationdb? I'd like to consume the FDB event log to do external inde…

Yes. I don't know how well documented it is, but there is an API (well, system keyspace) that can configure the database to log transactions for a selected key range (up to and including the whole database) into another selected key range. It is used by backup and asynchronous replication tools. The format of the configuration keys and transaction logs should be considered less stable than the core key/value API, whi…

> Alternatively, your application or layer can use the "versionstamp" atomic operations to write its own ordered log of what it is doing, or other indexing tricks.

I'm not sure I understand.

Are you suggesting having a second key space at `ops/{VERSIONSTAMP}` or something where values contain enough information about the operation to be able to process changes in an indexer? The indexer could then clean up after itself, deleting the operations once they had been ingested? ... Effectively using a portion of the keyspace as a queue?

Re: Apple open-sources FoundationDB

#284

Post by spullara » Fri, Apr 20, 2018, 1:52 AM UTC

tinco wrote: post #237

spullara wrote: post #217

Earlier quoted context omitted.

They had a document layer that wasn't released that was MongoDB compatible — so you can do this.

MongoDB is just a database with less features than a SQL database. An XML/XQuery database is fundamentally different, so I figured if FoundationDB layers are really so powerful, they might be able to model a tree DB as well.

Ah misread. Maybe? Storing hierarchical data would be pretty natural.

Re: Apple open-sources FoundationDB

#285

Post by yoda_sl » Fri, Apr 20, 2018, 1:56 AM UTC

dunham wrote: post #71

rudedogg wrote: post #56

Earlier quoted context omitted.

Maybe CloudKit? I was looking at the documentation for it recently and it's a key-value store.

CloudKit is using Cassandra/Solr: - http://www.vldb.org/pvldb/vol11/p540-shraer.pdf

From what I have heard, CloudKit is an API abstraction either of Cassandra/Solr and more recently Foundation DB.

So new CloudKit codebase nowadays is being run on top of Foundation DB

Re: Apple open-sources FoundationDB

#286

Post by mattkrea » Fri, Apr 20, 2018, 1:57 AM UTC

mercutio2 wrote: post #270

woolvalley wrote: post #115

Earlier quoted context omitted.

They are, slowly. Swift is open source, clang is open source. They are moving parts of the xcode IDE into open source, like with sourcekitd and now recently clangd. I don't think they will ever move 'secret sauce' into open source, but infrastructural things like DBs and dev tooling seems to be going in that direction.

Well, clangd is a Google project, which Apple has decided to start contributing to, so probably doesn’t belong on your list. Apple, like everyone else, wants to commoditize their complements.

Clang was an Apple project from the start.. I'm not sure what is telling you it is a Google project

Re: Apple open-sources FoundationDB

#287

Post by thx4thefish » Fri, Apr 20, 2018, 2:00 AM UTC

thx4thefish wrote: post #267

This seems nice. Besides a bunch of fanboy comments coming from the creators and devs, why is this exciting to the rest of us where things like the capability to join tables in an rdbms is trivial.

Thanks for the downvote. My question still remains.

Re: Apple open-sources FoundationDB

#288

Post by nl » Fri, Apr 20, 2018, 2:10 AM UTC

fizx wrote: post #211

voidmain wrote: post #194

Earlier quoted context omitted.

It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.

It'd take a low number of hours to wire up FoundationDB as a Lucene filesystem (Directory) implementation. Shared filesystem with a local RAM cache has been practical for a while in Lucene, and was briefly supported then deprecated in Elasticsearch. I've used Lucene on top of HDFS and S3 quite nicely. If you have a reason to use FoundationDB over HDFS, NFS, S3, etc, then this will work well. Doing a Lucene+DB impleme…

I wrote the original version of Solandra (which is/was Solr on Cassandra) on top of Jake's Lucene on Cassandra[1].

I can confirm it wasn't fast!

(And to be fair that wasn't the point - back then there were no distributed versions of Solr available so the idea of this was to solve the reliability/failover issue).

I wouldn't use it on a production system now days.

[1] http://nicklothian.com/blog/2009/10/27/solr-cassandra-soland...

Re: Apple open-sources FoundationDB

#289

Post by cakoose » Fri, Apr 20, 2018, 2:22 AM UTC

helper wrote: post #125

I'm very interested in hearing more about what running FoundationDB in production is like. I believe that FoundationDB stores rows in lexicographical order by key. Other databases like Cassandra strongly push you toward not storing data this way as it can easily lead to hotspots in the cluster. How do you deploy a FoundationDB cluster without leading to hotspots, or perhaps what operational actions are available to r…

Does Cassandra hash the primary key to get a more even distribution?

If you have a sorted data store, you can get the same distribution by keying off a hash of the "real" primary key, right?

Re: Apple open-sources FoundationDB

#290

Post by zbentley » Fri, Apr 20, 2018, 2:24 AM UTC

voidmain wrote: post #226

ddorian43 wrote: post #163

Earlier quoted context omitted.

Curious why the client is complicated compared to other dbs in same space ?

In some distributed databases the client just connects to some machine in the cluster and tells it what it wants to do. You pay the extra latency as it redirects these requests where they should go. In FDB's envisioned architecture, the "client" is usually a (stateless, higher layer) database node itself! So the client encompasses the first layer of distributed database technology, connects directly to services throu…

> but you had better have a good async RPC system!

The microservices crew with their "our database is behind a REST/Thrift/gRPC/FizzBuzzWhatnot microservice" pattern is still catching up to the significance of this statement.