This absolutely awesome!
I'm very excited for AntidoteDB, for its use cases but also for the underlying, pioneering work you're doing on CRDTs. Thank you for doing it! <3
Antidote: CRDT-based distributed database
71–80 of 89 posts
Re: Antidote: CRDT-based distributed database
#72Earlier quoted context omitted.
It's a government sponsored project so one would expect their publicity to be limited. That's part of the reason I posted it. Stuff like this is extremely exciting. It's at the cost of playing by the rules of CRDTs. Making CRDTs consequence-free is ongoing research.
> Making CRDTs consequence-free is ongoing research. What do you mean by that? I found possibly related language on this page the other day [0]: CRDTs are "[t]ypically not suited for editing application with consequent UI." Can you point me in the right direction? My googling got me nowhere. Thanks. [0] https://irisate.com/collaborative-editing-solutions-round-up...
It's not described in the docs, but elswhere [1] in their github repository you can find the note about rga, which is Replicated Growable Array CRDT, that can be used for building indexed linear sequences - shortly speaking arrays of elements with insert/remove semantics.
[1](https://github.com/SyncFree/antidote/blob/842874ca5ecffe947e...)
Re: Antidote: CRDT-based distributed database
#73There was also Datanet announced last year: http://highscalability.com/blog/2016/10/17/datanet-a-new-crd...
It's now rebranded as Kuhiro: http://highscalability.com/blog/2017/11/6/birth-of-the-nearc...
Re: Antidote: CRDT-based distributed database
#74There was also Datanet announced last year: http://highscalability.com/blog/2016/10/17/datanet-a-new-crd... It's now rebranded as Kuhiro: http://highscalability.com/blog/2017/11/6/birth-of-the-nearc...
There's also CRDB, a CRDT based version of Redis (closed source, and disclosure - I work for Redis Labs) https://redislabs.com/redis-enterprise-documentation/adminis...
Re: Antidote: CRDT-based distributed database
#75I wanted a database that would receive "events" asynchronously and stored that, but at the same time would process these events (from some piece of previously written code) to generate a queryable schema. If I wanted to change the schema later, the database would let me just rewrite the code and it would reprocess all the received events since the beggining. My use case is not anything high-performance or with thousa…
Don't you want a message queue like Sub/Pub?
Re: Antidote: CRDT-based distributed database
#76Earlier quoted context omitted.
Yes, but devil's in the details. The fact CRDTs are provably able to have specific consistency guarantees doesn't mean a particular implementation does it correctly in all contexts. Without testing, "We built this (city) on CRDTs" isn't very useful.
What's nice about building databases on compositions of CRDTs is that if you can validate the individual CRDTs via automated testing, you have a very high degree of confidence the composition of those CRDTs will do something similar. No one's arguing that a Jepsen test shouldn't be done. Just that it'll probably be very different in character from more invented industry technology.
Yes; if you can prove a system was built atop an academically sound paradigm, it's more likely to adhere to the guarantees that paradigm is intended to provide than a system built atop an unproven paradigm. Kinda goes without saying, though. :P
Unsnarkily, I agree that implementation and composition of CRDTs is comparatively straightforward compared with other approaches people have taken. But if correct behavior is a requirement, full testing is too, regardless of how easy it -should- be in theory.
Re: Antidote: CRDT-based distributed database
#77At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…
(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.
> However, it's largely an engineering resource problem and not a theoretical problem.
I appreciate your openness and honesty here, but the moment that an academic says that all the theory is solved and now "it's largely an engineering resource problem" is when people from industry tend to get nervous :-)
The majority of aspects that make databases robust aren't theoretical problems but "mere" engineering problems. Nevertheless there's an enormous variety in quality in this area. Pile sufficiently many engineering problems together and it becomes very hard to get right.
Re: Antidote: CRDT-based distributed database
#78At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…
(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.
CRDTs accumulate garbage, and need a global "sync" to GC. How do you mitigate this from having performance impacts?
Re: Antidote: CRDT-based distributed database
#79At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…
(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.
I think you know that’s a cliche in some circles and my intention is not to try call you on that without reason, but to ask about it practically speaking.
Would you agree that it seems for many disciplines, whether its cs or physics, the power of the statement “it’s solved theoretically” relates to the scope of the problem?
For example, if someone came up with a strong argument or proof that raises the upper bound of performance for a specific algorithm, I could be convinced to start celebrating right away, and would think nothing of the implementation being passed off to a student as a mere formality.
However, the scope doesn’t have to increase much before a formal proof, or even a conclusive argument, become impossible to make in a way that’s completely convincing.
It’s not a knock against either theory or engineering, it’s a simple matter of our inability to model or predict things accurately beyond a certain complexity.
I only wish I could explain this well to lay people, like if a friend asks me, “after over half a century of writing code why cant even the greatest software companies in the world reliably predict how long it takes to ship software?”
Maybe next time I’ll tell them, “it’s the same reason we need database reference implementations”. It will just add more confusion, but at least I can spit out some kind of answer.
Re: Antidote: CRDT-based distributed database
#80Earlier quoted context omitted.
(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.
> However, it's largely an engineering resource problem and not a theoretical problem. I appreciate your openness and honesty here, but the moment that an academic says that all the theory is solved and now "it's largely an engineering resource problem" is when people from industry tend to get nervous :-) The majority of aspects that make databases robust aren't theoretical problems but "mere" engineering problems. N…
Sure, but if someone (or some company, rather) really cares, the "mere" engineering problems in an (early-stage, not-yet-a-disaster-of-a) codebase can be solved by throwing money at them—to hire, or contract, the best engineers.
But fundamental "academic" problems in the system architecture won't ever be solved, because the resulting codebase would be a different product targeting different use-cases.