FoundationDB: A distributed, unbundled, transactional key value store [pdf] • Hacker News Board

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#101

Post by cwp » Thu, Jun 10, 2021, 8:33 PM UTC

jen20 wrote: post #86

cwp wrote: post #56

Earlier quoted context omitted.

Sure quibble about the details. The point is, these are all attempts to make PG more scalable. Around the time these projects got started Foundation looked like abandonware. If it hadn't, it's possible that "how can we have really scalable SQL databases?" might have had Foundation as part of the answer.

FoundationDB was proprietary software before the acquisition, and not open source, so I’d say close to “no chance”.

Good point. Still, if FDB was high-profile proprietary software during this time it might have inspired an open source clone. Spanner inspired Cockroach and Yugabyte, after all.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#102

Post by cre-ker » Mon, Jun 14, 2021, 9:43 PM UTC

tpetry wrote: post #79

davgoldin wrote: post #78

Earlier quoted context omitted.

FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte coun…

What kind of read/write ratio are you using? And would your solution work for a write-heavy workload? Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs…

Kafka limits are customizable. 10MB is quite small. Kafka easily handles 100MB messages. If you need some temporal storage that works like a log then there's nothing better than kafka in terms of speed and scalability. But it's more about relatively long term storage. If you need to persist data for very short amount of time, why don't you use in-memory store? Do you have strict requirements around data loss? Something like Redis much just do the trick for out. With Redis cluster it even scales.

Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]

#103

Post by refset » Wed, Jun 16, 2021, 7:33 PM UTC

jwr wrote: post #96

refset wrote: post #90

Earlier quoted context omitted.

Hi, I work on the Crux team. I think "fully distributed" has a few possible meanings, but is it essentially a case of wanting something with a dead-simple clustering story? Or is it more about multi-region distribution & availability? Whilst Kafka itself is almost certainly not as simple to operate as FDB (although I can't speak from experience), it does in turn provide Crux with dead-simple clustering, because each…

I'm sorry — I tried to respond quickly, and this kind of response always lacks depth. I wasn't criticizing Crux by any means. I like the project, and I spent a long time thinking carefully about bitemporality, as this is something I often need and have to implement by myself. My decision to go with FDB was based on many factors, and it was taken over the course of multiple years. Some factors were technical (e.g. ful…

No need to apologise at all, and thank you for your thoughtful reply!

>To get the full value out of it, your app code should be aware of transactions, participate in database mechanisms (versionstamps), correctly handle asynchronous streaming of large amounts of data with backpressure

This is an excellent point, and broadly aligns with Crux's design also, since the transaction semantics provided are relatively raw and therefore much of the interesting parts of the "database" logic live firmly in the application code.

It sounds like you have a really interesting system and I would love to hear more about it one day :)