Does anyone know of the"sqlite connector" mentioned in this post ?
https://opensourceconnections.com/blog/2013/05/06/does-found...
It would be really cool to find it, if it's still out there.
FoundationDB: A distributed, unbundled, transactional key value store [pdf]
81–90 of 103 posts
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#82>> In its newest release, CouchDB [2] (arguably the firstNoSQL system) is being re-built as a layer on top of FoundationDB.
That is impressive. Like a framework for implementing noSQL DBs.
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#83This is a really good document:
https://apple.github.io/foundationdb/data-modeling.html
I have been studying these key-value stores with efficient range iteration lately (such as LevelDB, RocksDB, BigTable, FoundationDB, etc). This is a great reference on how to make such a simple abstraction do a lot of useful things.
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#84Earlier quoted context omitted.
That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).
Also, would it make any sense to use FoundationDB instead of Kafka as Document Store for Crux?
This is actively being discussed in #crux on Clojurians Slack [1].
It ought to show up within the archive [2] in a couple of days. Should already be in the Clojurians Zulip mirror as well.
[1]: https://clojurians.slack.com/archives/CG3AM2F7V/p16231025131...
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#85Earlier quoted context omitted.
Those are two completely non overlapping use cases. If you can use pgsql for your problem, you have no business trying to use a distributed key value store instead. That would be at least as dumb as driving screws with a hammer.
Yeah, but there are quite a few efforts out there to extend PG into a distributed DB of one flavor or another. Some examples are YugabyteDB, CockroachDB, Aurora and Citus. It's a reasonable approach, but it's also reasonable to come at it from the other direction - build a SQL engine on top of a solid distributed key-value store. Contrafactuals are always dicey, but FDB vanishing behind the Apple wall of silence sure…
CockroachDB is exactly this - a SQL engine on top of a distributed key value store. It is not an extension of Postgres itself, it just speaks the protocol and implements many of the features.
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#86Earlier quoted context omitted.
> Some examples are YugabyteDB, CockroachDB, Aurora and Citus. Of those the first two are not PG, they just share the wire protocol and try to be compatible at a SQL level. Aurora is not really distributed, it's replicated for availability and durability six ways at the block storage level. Citus is distributed as I understand it though. > but it's also reasonable to come at it from the other direction - build a SQL…
Sure quibble about the details. The point is, these are all attempts to make PG more scalable. Around the time these projects got started Foundation looked like abandonware. If it hadn't, it's possible that "how can we have really scalable SQL databases?" might have had Foundation as part of the answer.
FoundationDB was proprietary software before the acquisition, and not open source, so I’d say close to “no chance”.
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#87Have nothing but praise for FoundationDB. It has been by far the most rock solid distributed database I have ever had the pleasure of using. I used to manage HBase clusters, and the fact that I have never once had to worry about manually splitting "regions" is such a boon for administration...let alone JVM GC tuning. We run several FDB clusters using 3-DC replication and have never once lost data. I remember when we…
How would it compare to say something like hosted Redis, or if you wanna be more fancy ElasticSearch. I have been looking into FDB for pretty long time and have been looking for a perfect opportunity to use it. Would be helpful if you can describe your usage scenario (kind of data you are storing).
The key takeaways of FoundationDB is that it is a strongly consistent KeyValue store that preserves ordering (lexicographical). Although you might consider FDB's APIs to be quite primitive (get/set/scan), the payoff is how it seamlessly handles multi-key transactions without requiring you to write or manage some client side two-phase-commit process.
Given these primitives, you or other engineers can write higher level APIs on top (e.g. sql, search, etc). In fact, we make extensive use of their RecordLayer [1] library which provides a strongly consistent schema based write process using Protobuf. This includes on-write-consistent indexes. Apple has also open-sourced a MongoDB API [2] compliant interface that allows you to get all the consistency guarantees from FDB, but with an API interaction of MongoDB.
The beauty of FDB is that the primitives are done in such a rock solid fashion, you can write higher level APIs without having to worry about the really hard stuff (transactions, failures, config errors, testing, simulation, etc). Another example is that CouchDB is switching their back-end to use FDB for their 4.0 release.
Given that the database sorts data lexicographical...to us, it became a natural fit for an online (always-mutable) time-series database.
If you want more insight in how we use it, I go over it in some detail towards the end of my keynote [3] from last August.
[1] https://www.foundationdb.org/blog/announcing-record-layer/
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#88I just implemented a database with changefeeds using FoundationDB (in Clojure), to eventually replace RethinkDB in my system. Very impressed so far.
That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).
Yes, I did. But I had specific requirements, and the main one was that I need a fully distributed database, where one of the nodes can disappear for any reason at any time and things would just continue as if nothing happened.
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#89Earlier quoted context omitted.
FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed. The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance. For statistics like folder byte coun…
What kind of read/write ratio are you using? And would your solution work for a write-heavy workload? Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs…
From own experience, I'd say FDB can handle it all, we've got 20% read 80 write during peak hours, and reverse: 80 read 20 write other time.
Without doing your own tests, here's per core numbers, from which you can extrapolate (e.g. via CPU mips) towards your own hardware: https://apple.github.io/foundationdb/performance.html#throug...
FDB is ACID as is shipped, you don't need to turn knobs to make it such. Toughest part is to figure out classes/roles of the system. Here are a couple of good starting points: https://nikita.melkozerov.dev/posts/2019/06/building-a-found... https://forums.foundationdb.org/t/roles-classes-matrix/1340/...
Re: FoundationDB: A distributed, unbundled, transactional key value store [pdf]
#90Earlier quoted context omitted.
That’s awesome! I’m interested specifically in using FDB with Clojure. Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).
Yes, I did. But I had specific requirements, and the main one was that I need a fully distributed database, where one of the nodes can disappear for any reason at any time and things would just continue as if nothing happened.
Hi, I work on the Crux team. I think "fully distributed" has a few possible meanings, but is it essentially a case of wanting something with a dead-simple clustering story? Or is it more about multi-region distribution & availability?
Whilst Kafka itself is almost certainly not as simple to operate as FDB (although I can't speak from experience), it does in turn provide Crux with dead-simple clustering, because each Crux node acts as an isolated replica. FDB could be used equally instead of Kafka, but the maximum write throughput would then be somewhat lower. The only key part of the story Crux doesn't currently supply out-of-the-box is a load-balancing layer atop a cluster of nodes.
I expect the real drawback of Crux's current design, by comparison, is that each node is an ~expensive full replica, whereas your system (built on FDB) will benefit from fully sharded indexes running across a cluster of (smaller) machines. The main trade-off then is the low-latency query performance of a fully local KV store.