Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple open-sources FoundationDB

foundationdb.org

Post reply on HN
Print view View original

381–390 of 453 posts

Re: Apple open-sources FoundationDB

#381

Post by helper »

cakoose wrote: post #289
helper wrote: post #125

I'm very interested in hearing more about what running FoundationDB in production is like. I believe that FoundationDB stores rows in lexicographical order by key. Other databases like Cassandra strongly push you toward not storing data this way as it can easily lead to hotspots in the cluster. How do you deploy a FoundationDB cluster without leading to hotspots, or perhaps what operational actions are available to r…

Does Cassandra hash the primary key to get a more even distribution? If you have a sorted data store, you can get the same distribution by keying off a hash of the "real" primary key, right?

Cassandra allows you to configure a "partitioner" that determines which nodes a primary key belongs to. There is a ByteOrderPartitioner that stores partitions in order lexicographically by primary key. There is also a Murmor3 hash based partitioner (which is the recommended default).

Cassandra allows you to store multiple records in sorted order within a partition. The normal recommended way to get data locality is to store records that are frequently accessed together in the same partition.

Re: Apple open-sources FoundationDB

#382

Post by pdeva1 »

sheeshkebab wrote: post #143

It’s neat although there is no sql front end. Bloomberg’s comdb2 was open sourced recently https://github.com/bloomberg/comdb2 - it seems similar, but would be interesting to see comparison.

comdb2 is not a big data db per say. it compares more to mysql than foundationdb https://blog.dripstat.com/first-look-at-bloombergs-amazing-c...

Re: Apple open-sources FoundationDB

#383

Post by deepanchor »

pacala wrote: post #317
deepanchor wrote: post #298

Earlier quoted context omitted.

Didn't downvote you, but I believe the excitement is over FoundationDB's ability to perform ACID compliant distributed transactions without sacrificing performance -which to my knowledge no current RDBMS or even NoSql can do.

> no current RDBMS can do https://cloud.google.com/spanner/ ?

Meant to say "no current open-source RDBMS"

Re: Apple open-sources FoundationDB

#384

Post by wejick »

Never heard about fdb but the concept is very intriguing. We know many nosql db on top kv store rocksdb or level db (like dgraph before moving to new storage engine). That it still needs more to write until it can be called distributed and scalable. By using foundation db we can skip many parts and focus on other part like query language and API. That's why I like the layer concept, unfortunately the is very little documentation about it. Found some layer written in python in the repo, but I don't understand where is the position of layer in the general architecture. I thought it would like plugin but I think that's not the case.

Re: Apple open-sources FoundationDB

#385

Post by jwatte »

voidmain wrote: post #207
jamesblonde wrote: post #179

Earlier quoted context omitted.

Thanks for the detailed answer. Is it actually serializable isolation - does it handle write skew anomalies ( https://en.wikipedia.org/wiki/Snapshot_isolation )? Most OCC systems I know have only snapshot isolation. Systems that sound closest to FoundationDB's transaction model that i can think of are Omid ( https://omid.incubator.apache.org/ ) and Phoenix ( https://phoenix.apache.org/transactions.html ). They both s…

Yes, it's really serializable isolation. The real kind, not the "doesn't exhibit any of the named anomalies in the ANSI spec" kind. We can selectively relax isolation (to snapshot) on a per-read basis (by just not creating a conflict range for that read). I tried to explain distributed resolution elsewhere in the thread. I believe our approach to indices pretty much totally dominates per-partition indexing. You can e…

My guess is the objection lies in "Have to manage the index myself."

Also, the main draw-back of "indices as data" in NoSQL is when you need to add a new index -- suddenly, you have to scrub all your data and add it to the new index, using some manual walk-the-data function, and you have to make sure that all operations that take place while you're doing this are also aware of the new index and its possibly incomplete state.

Certainly not impossible to do, but it sometimes feels a little bit like "I wanted a house, but I got a pile of drywall and 2x4 framing studs."

Re: Apple open-sources FoundationDB

#386

Post by ile »

It is strange, why ArangoDB isn't mentioned often on HN, or in this thread. It is multi-model (KV, document, graph) database (with transactions) and I have been happily using it for a while now.

Haven't scaled it yet to any large installations, so I can speak about how well it does that.

Re: Apple open-sources FoundationDB

#388

Post by voidmain »

lmorris84 wrote: post #337
nlavezzo wrote: post #34

Earlier quoted context omitted.

Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it's fantastic that it's available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.

I wrote a java port of the python counter many moons ago [1]. Will have to resurrect it! [1] https://github.com/leemorrisdev/foundationcounter

It should probably be pointed out that atomic increment is in most situations a more efficient solution for high contention counters in modern FDB.

Re: Apple open-sources FoundationDB

#389

Post by voidmain »

jwatte wrote: post #385
voidmain wrote: post #207

Earlier quoted context omitted.

Yes, it's really serializable isolation. The real kind, not the "doesn't exhibit any of the named anomalies in the ANSI spec" kind. We can selectively relax isolation (to snapshot) on a per-read basis (by just not creating a conflict range for that read). I tried to explain distributed resolution elsewhere in the thread. I believe our approach to indices pretty much totally dominates per-partition indexing. You can e…

My guess is the objection lies in "Have to manage the index myself." Also, the main draw-back of "indices as data" in NoSQL is when you need to add a new index -- suddenly, you have to scrub all your data and add it to the new index, using some manual walk-the-data function, and you have to make sure that all operations that take place while you're doing this are also aware of the new index and its possibly incomplet…

"I wanted a house, but I got a pile of drywall and 2x4 framing studs."

This is a totally legitimate complaint about FoundationDB, which is designed specifically to be, well, a foundation rather than a house. If you try to live in just a foundation you are going to find it modestly inconvenient. (But try building a house on top of another house and you will really regret it!)

The solution is of course to use a higher level database or databases suitable to your needs which are built on FDB, and drop down to the key value level only for your stickiest problems.

Unfortunately Apple didn't release any such to the public so far. So I hope the community is up to the job of building a few :-)

Re: Apple open-sources FoundationDB

#390

Post by lmorris84 »

voidmain wrote: post #388
lmorris84 wrote: post #337

Earlier quoted context omitted.

I wrote a java port of the python counter many moons ago [1]. Will have to resurrect it! [1] https://github.com/leemorrisdev/foundationcounter

It should probably be pointed out that atomic increment is in most situations a more efficient solution for high contention counters in modern FDB.

Ah I don’t believe that was available last time I used it - I’ll check it out thanks!
Post reply on HN