Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FoundationDB: A distributed, unbundled, transactional key value store [pdf]

foundationdb.org

Post reply on HN
Print view View original

71–80 of 103 posts

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#71

Post by jFriedensreich »

jng wrote: post #54
jFriedensreich wrote: post #47

Earlier quoted context omitted.

its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for…

Lucene is Java, right? There should be space for a native implementation, like ScyllaDB is doing to Cassandra (and DynamoDB, though the gap is not of the same shape in the last case). Or am I missing something? I used ElasticSearch and run one cluster in production in the past and found it horrible. Maybe I'm missing something, since they're so successful even as a public company...

just out of interest, what was horrible about es?

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#72

Post by jFriedensreich »

mirker wrote: post #57
jFriedensreich wrote: post #46

Earlier quoted context omitted.

large unstructured blobs and large files are among the things not well suited to foundationdb and couchdb 4 actually reduced supported blob size in the transition to foundationdb. it looks like object/blob storage systems are at the moment rather seperating more from key/value and document storage than growing together. but this is a good thing because the tradeoffs are very different and it allows each system to foc…

Can you elaborate on what requirements these blob systems should have? My understanding is object stores are typically “flat” by design to scale well (in contrast to a tree structure found in filenames). For content addressing, are people using the keys in sophisticated ways or are the values being indexed? Any reason to push this complexity into the storage layer as opposed to composing the functionality?

well there is often no hierarchy as in folders on an old school filesystem but if the system uses chunking, the organization of blob chunks is very important for the performance and scaling characteristics. The chunking algorithm needs to be performant but also lead to sensible chunk size and count and in addition can also do data based boundaries so chunks can be reused even if blob data changes at the start of the data. This can be different depending on your specific application (eg. the read/write ratio and average file sizes) and requirements for optimal use of the underlying filesystem, that's one reason why no de facto standard chunker has been established so far. There are many tradeoffs for key organization too. Do you need more sophisticated range queries or only single keys? How balanced is growing and shrinking of your data structure vs performance? What is the clustering story? How do you handle rebalancing/cleanup/pruning? Is your primary key organization content hashes like in ipfs or more arbitrary strings as in s3/minio? Is your metadata/ secondary keys system completely integrated or more independent?

Thats exactly what your last questions points to. If you are lets say dropbox and have probably a super sophisticated key value store setup i can imagine you would want your content addressable layer to be as simple and narrowly optimized as possible and develop and optimize the indexing, metadata and key queries system nearly completely separately. If you are working on some system that also should scale down to run on individual machines like ipfs, git annex or minio before their focus on kubernetes you want a system that can run as a single daemon but also where users can reason about the whole system as an integrated concept.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#73

Post by krn »

adamfeldman wrote: post #55
jwr wrote: post #36

I just implemented a database with changefeeds using FoundationDB (in Clojure), to eventually replace RethinkDB in my system. Very impressed so far.

That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).

Also, would it make any sense to use FoundationDB instead of Kafka as Document Store for Crux?

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#75

Post by bpicolo »

jFriedensreich wrote: post #47
agency wrote: post #43

Earlier quoted context omitted.

I’m curious about this as well. Is anyone working on building text search on top of FDB? It’s kind of astounding to me that last time I checked Elasticsearch was still essentially the only game in town.

its pretty hard to catch up with lucene, there is just so much work, features and brainpower in there at this point. as many features of foundationdb such as the transaction guarantees and reliability are not super important for fulltext search i cannot imagine any company even apple or ibm being able to justify that gigantic investment, instead im sure nearly any soluion willcontinue to use lucene under the hood for…

Tantivy is giving it a good go: https://github.com/tantivy-search/tantivy

The good thing about Lucerne existing is you’re allowed to also use their good ideas.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#76

Post by jamesblonde »

davgoldin wrote: post #67

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HD…

Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files (We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution?

(Discussed here on HN: https://news.ycombinator.com/item?id=25149154 )

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#77

Post by jamesblonde »

jamesblonde wrote: post #76
davgoldin wrote: post #67

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HD…

Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files ( We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution? (Discussed here on HN: https://news.ycombinator.com/item?id=25149154 )

Do you support atomic rename? Atomic rename on subtrees? Consistent directory listings?

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#78

Post by davgoldin »

tpetry wrote: post #68
davgoldin wrote: post #67

Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system. We've tried Ceph, GlusterFS, HD…

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.

FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed.

The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance.

For statistics like folder byte count, you could use FDB's atomic increment operation.

Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundation...

I'll be happy to answer specific question if you got any.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#79

Post by tpetry »

davgoldin wrote: post #78
tpetry wrote: post #68

Earlier quoted context omitted.

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.

FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte coun…

What kind of read/write ratio are you using? And would your solution work for a write-heavy workload?

Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs tuneable for acid? Perfect acid requirements would not really be needed, if everything is fsynced every second would be completely ok.

Post reply on HN