331–340 of 453 posts
Earlier quoted context omitted.
How would you replace a Lucene/Elasticsearch index with foundationDb?
It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.
Earlier quoted context omitted.
I think you are on the right track. Storing every individual (term, document, ...) in the key value store will not be efficient, but you should be able to take Lucene's nice fast immutable data structure and stuff blocks of it (at the term level or below) into FDB values very efficiently. And of course you can do caching (and represent invalidation data structures in FDB), and... FDB leaves room for a lot of creativi…
> you should be able to take Lucene's nice fast immutable data structure and stuff blocks of it (at the term level or below) into FDB values very efficiently. That sounds a lot like Datomic's "Storage Resource" approach, too! Would Datomic-on-FDB make sense, or is there a duplication of effort there?
Datomic’s single-writer system requires conditional put (CAS) for index and (transaction) log (trees) roots pointers (mutable writes), and eventual consistency for all other writes (immutable writes) [0].
I would go as far as saying a FoundationDB-specific Datomic may be able to drop its single-writer system due to FoundationDB’s external consistency and causality guarantees [1], drop its 64bit integer-based keys to take advantage of FoundationDB range reads [2], drop its memcached layer due to FoundationDB’s distributed caching [3], use FoundationsDB watches for transactor messaging and tx-report-queue function [4], use FoundationDB snapshot reads [5] for its immutable indexes trees nodes, and maybe more?
Datomic is a FoundationDB layer. It just doesn’t know yet.
[0] https://docs.datomic.com/on-prem/acid.html#how-it-works
[1] https://apple.github.io/foundationdb/developer-guide.html?hi...
[2] https://apple.github.io/foundationdb/developer-guide.html?hi...
[3] https://apple.github.io/foundationdb/features.html#distribut...
[4] https://docs.datomic.com/on-prem/clojure/index.html#datomic....
[5] https://apple.github.io/foundationdb/developer-guide.html?hi...
Earlier quoted context omitted.
I'm not familiar with FDB but what you say sounds almost too good to be true. Can I use it to implement the Google Datastore api? I'm trying for years to find a suitable backend so that I can leave the Google land. Everything I tried either required a schema or lacked transactions or key namespaces.
Hey there--I'm an engineer on the Cloud Datastore team. I'd love to know more about what your needs are if you're willing to share.
Earlier quoted context omitted.
CockroachDB (and I assume TiDB as well) can have the same properties and same awesome latencies as Spanner, if you have the same hardware. So if you actually put down your own fiber, and install GPS clocks in each rack, you’ll be able to enjoy the same results.
Does CockroachDB use TrueTime or a similar technique?
[1] https://cloud.google.com/spanner/docs/true-time-external-con...
[2] https://static.googleusercontent.com/media/research.google.c...
Earlier quoted context omitted.
Spanner can do global consistency and (some?) transactions but I'm unaware of it being able to do the sort of layering Foundation can internally to expose largely different database forms on top of it. I have never used Spanner, though, so I'm open to being corrected.
What do you mean by “forms”? Spanner is also layered. The bottom layer is basically a key-value store. On top of that there’s a full blown SQL layer, which, BTW can work with hierarchical records as well as flat tables. Both support transactions and guarantee global consistency.
Earlier quoted context omitted.
It's more like you would build a better Elasticsearch using Lucene to do the indexing and FoundationDB to do the storage. FoundationDB will make it fault tolerant and scalable; the other pieces will be stateless.
ok thanks, it was sort of confusing me
Earlier quoted context omitted.
Will said what I wanted to say, but: me too. I'm super happy about this and grateful to the team that made it happen! (I was one of the co-founders of FoundationDB-the-company and was the architect of the product for a long time. Now that it's open source, I can rejoin the community!)
Another (non-technical) founder here - and I echo everything voidmain just said. We built a product that is unmatched in so many important ways, and it's fantastic that it's available to the world again. Will be exciting to watch a community grow around it - this is a product that can benefit hugely from OS contributions as layers that sit on top of the core KV store.
I can see everyone's extremely happy about this, which is great. As someone who's never used it, I'd like to know more about FoundationDB and how it compares to other offerings such as MySQL or Postgres, and which use cases is it most suited to. I would especially love to hear the thoughts of those with direct experience of using Foundation DB. Thanks!
Personally I'd be more interested in hearing how this compares to other distributed noSQL implementations like Cassandra.
Earlier quoted context omitted.
Thanks for the downvote. My question still remains.
Didn't downvote you, but I believe the excitement is over FoundationDB's ability to perform ACID compliant distributed transactions without sacrificing performance -which to my knowledge no current RDBMS or even NoSql can do.
I hadn't heard of FoundationDB before, so I did some digging into the features: https://apple.github.io/foundationdb/features.html . It seems to claim ACID transactions with serializable isolation, but also says later on that it uses MVCC, slower clients won't slow down operations, and that it allows true interactive queries. I didn't think an MVCC implementation could provide that level of isolation, and I'm not eve…
I'll try to give you a quick introduction. The architecture talk I recorded for new engineers working on the product ran to four or five hours, I think :-). In short, it is serializable optimistic MVCC concurrency. A FDB transaction roughly works like this, from the client's perspective: 1. Ask the distributed database for an appropriate (externally consistent) read version for the transaction 2. Do reads from a cons…