Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB: A distributed, unbundled, transactional key value store [pdf]

foundationdb.org

Post reply on HN
Print view View original

91–100 of 103 posts

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#91

Post by mirker »

jFriedensreich wrote: post #72
mirker wrote: post #57

Earlier quoted context omitted.

Can you elaborate on what requirements these blob systems should have? My understanding is object stores are typically “flat” by design to scale well (in contrast to a tree structure found in filenames). For content addressing, are people using the keys in sophisticated ways or are the values being indexed? Any reason to push this complexity into the storage layer as opposed to composing the functionality?

well there is often no hierarchy as in folders on an old school filesystem but if the system uses chunking, the organization of blob chunks is very important for the performance and scaling characteristics. The chunking algorithm needs to be performant but also lead to sensible chunk size and count and in addition can also do data based boundaries so chunks can be reused even if blob data changes at the start of the…

I see. Some of these, like ipfs, are more general purpose systems (basically communication protocols) than I was thinking of.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#93

Post by mping »

twoodfin wrote: post #44

Did they ever implement a SQL layer? They seemed like one of the only NoSQL products with the architecture to make it plausible to do so.

There was an SQL layer but performance was sub-par IIRC. There was also a blog post somewhere explaining why it's probably not a good idea to build an SQL layer on top of a KV store, devil in details, etc.

Edit: found it https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas...

Not sure what to think of it, I'm not a DB expert by any means but the post sounds plausible enough and the SQL layer is discontinued AFAIK. I guess with each new abstraction layer you leave some perf on the table.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#94

Post by foobiekr »

Meai wrote: post #63
__float wrote: post #62

Earlier quoted context omitted.

https://apple.github.io/foundationdb/administration.html#rem... What is this process describing, and how does it differ from what you were trying to do?

They are using the term "machine" and "process" in multiple conflicting meanings I think. I mean maybe they improved this since I used it 1.5 years ago but I kind of doubt it. If I remember correctly there is a definite problem if you remove one of the processes that end up guaranteeing your configured redundancy mode. So then suddenly your entire cluster is inoperable. Yes really, I think it doesn't even properly re…

This is not true.

We've run fdb in production for several years at this point. We have dramatically scaled production clusters up and down.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#95

Post by __float »

one2three4 wrote: post #82

>> In its newest release, CouchDB [2] (arguably the firstNoSQL system) is being re-built as a layer on top of FoundationDB. That is impressive. Like a framework for implementing noSQL DBs.

This is similar to WiredTiger, and its role in MongoDB.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#96

Post by jwr »

refset wrote: post #90
jwr wrote: post #88

Earlier quoted context omitted.

Yes, I did. But I had specific requirements, and the main one was that I need a fully distributed database, where one of the nodes can disappear for any reason at any time and things would just continue as if nothing happened.

Hi, I work on the Crux team. I think "fully distributed" has a few possible meanings, but is it essentially a case of wanting something with a dead-simple clustering story? Or is it more about multi-region distribution & availability? Whilst Kafka itself is almost certainly not as simple to operate as FDB (although I can't speak from experience), it does in turn provide Crux with dead-simple clustering, because each…

I'm sorry — I tried to respond quickly, and this kind of response always lacks depth.

I wasn't criticizing Crux by any means. I like the project, and I spent a long time thinking carefully about bitemporality, as this is something I often need and have to implement by myself.

My decision to go with FDB was based on many factors, and it was taken over the course of multiple years. Some factors were technical (e.g. fully distributed, correct, ability to implement changefeeds) and some were not (I am not happy with the internal complexity of Kafka).

Also, I don't feel comfortable with the way most DB discussions are framed. Most people think that one can choose a database and switch between databases at will, which implies there is a clearly defined division between "a database" (with an API) and "the app". That is not necessarily true with FDB. To get the full value out of it, your app code should be aware of transactions, participate in database mechanisms (versionstamps), correctly handle asynchronous streaming of large amounts of data with backpressure, etc.

In case of my app, I did not want to switch from one "document" database to another, I wanted to be able to write a distributed application based on an underlying well-tested transactional database. That's why FDB is a good fit, and by "FDB" I do not mean their document db layer, I mean the basic FDB itself (I only use the tuple layer).

I hope that explains things a bit more.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#97

Post by jng »

jFriedensreich wrote: post #71
jng wrote: post #54

Earlier quoted context omitted.

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

just out of interest, what was horrible about es?

It was brittle beyond belief. The process would run out of memory and crash really easily. You are supposed to dimension resources well, but of course you'd like to handle things more gracefully when a huge inflow of data or requests comes in, good engineering is "graceful failure" such as just rejecting requests, instead of total mayhem and destruction. It was "interesting" to discover that the monitoring plugins included by default would bring down the entire cluster. 5 beefy AWS machines brought down when pointing their own monitor at them! Yes there was significant input traffic, but I would expect more solid behavior from production-ready software.

Shard migration and duplication was a beauty to watch when it worked well, but very often it wouldn't. I tried to establish a procedure to bring instances back up safely and repeatably, but sometimes I would just have to try and try again, no procedure guaranteed restoring operation in all cases. I do remember spending more than one weekend babysitting that cluster. There's a special place in my heart for that memory, and it's not a beautiful one.

I was very surprised to see them IPO successfully - this test was before they were public. I assumed the open source version was just missing the kind of tools and know how to be able to work solidly without paying them or their consultants. These days, I think it's poor engineering resulting in brittleness, we shouldn't have put so much traffic to it, and probably everyone out there experiences the same behavior unless they ensure much safer margins, which probably makes it expensive but I guess who's complaining.

I ended up designing and writing my own (much simpler) distributed system for what I needed (much simpler than everything ElasticSearch does, of course), and it's been in reliable operation since summer 2015. Good engineering goes a long way. Now in the process of turning that approach into a startup. If I get 1/10th as successful as ES I will be a happy camper :)

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#98

Post by PeterCorless »

jng wrote: post #54
jFriedensreich wrote: post #47

Earlier quoted context omitted.

its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for…

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

Tantivy? Written in Rust. More like Lucene than Elastic, though.

https://github.com/tantivy-search/tantivy

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#99

Post by PeterCorless »

jFriedensreich wrote: post #69
vosper wrote: post #59

Earlier quoted context omitted.

There’s a lot to Lucene, and there’s also a lot to ElasticSearch. And I think they’re fairly tightly coupled. But I do think that a well-funded and skilled startup team could take a run at ES. They’re a monopoly in their niche. There has to be money in disrupting them.

im sure its possible in theory, but without a permissive license it would be not relevant to most applications that are interesting. now try convincing a venture backed startup to build a lucene alternative and license it permissively after seeing what happend with amazon and es...

Tantivy (written in Rust) uses MIT. I can't speak to the quality of the code, but their aim is to rewrite Lucene-type functionality as a Rust crate.

https://github.com/tantivy-search/tantivy

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#100

Post by voidmain »

mping wrote: post #93
twoodfin wrote: post #44

Did they ever implement a SQL layer? They seemed like one of the only NoSQL products with the architecture to make it plausible to do so.

There was an SQL layer but performance was sub-par IIRC. There was also a blog post somewhere explaining why it's probably not a good idea to build an SQL layer on top of a KV store, devil in details, etc. Edit: found it https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas... Not sure what to think of it, I'm not a DB expert by any means but the post sounds plausible enough and the SQL layer is discontinued A…

It's not a very good analysis. The only reason there isn't a good OLTP SQL engine on top of FDB is that no one has invested the considerable resources and expertise to build one. The FDB SQL Layer was on its way, but Apple prioritized other things. Snowflake built a world class analytic SQL database on FDB and the door is still open for someone to do OLTP. There might even be room for two different designs (one more like Akiban/SQL Layer or F1, competing with Spanner, and one more like Aurora, with a more familiar performance envelope but slightly less scalability)
Post reply on HN