Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antidote: CRDT-based distributed database

syncfree.github.io

Post reply on HN
Print view View original

81–89 of 89 posts

Re: Antidote: CRDT-based distributed database

#81

Post by anne_biene »

runT1ME wrote: post #78
cmeiklejohn wrote: post #67

Earlier quoted context omitted.

(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.

CRDTs accumulate garbage, and need a global "sync" to GC. How do you mitigate this from having performance impacts?

[Disclaimer: I am an Antidote maintainer]

Some CRDTs support garbage collection directly - if you run them in a causally consistent environment. Antidote is causally consistent and has a Set and Map implementation that work like this; for these CRDTs you don't need a global sync.

Re: Antidote: CRDT-based distributed database

#82

Post by jpdsousa »

lopatin wrote: post #21
gjem97 wrote: post #7

Earlier quoted context omitted.

I think the issue you're looking for is a limited number of supported datatypes. This doesn't appear to be a general-purpose database at the moment: don't expect to drop it in as a replacement for MySQL. Doesn't mean it can't be in the future, but it would have to fall back on other techniques for the datatypes that can't be built with CRDTs. http://syncfree.github.io/antidote/crdts.html

I think the data types look fine. If I'm using such a database, I don't think I want the CRDT data types to be abstracted away behind some general-purpose (SQL?) interface. Can that even be done? I'd rather just pay the small price of restructuring my data into maps and sets and have the CRDT database do the rest without any extra indirection.

Well, i've been working on an SQL interface for AntidoteDB for my master thesis, and yes, it can be done. However, there are some limitations. At this point we are mostly focusing in constraints (i.e. invariant preservation), and we've already figured out stuff like numeric bounds, entity integrity and referential integrity (this last one is quite interesting IMO). Github: https://github.com/JPDSousa/AQL

Re: Antidote: CRDT-based distributed database

#83

Post by marc_shapiro »

sturadnidge wrote: post #66
cmeiklejohn wrote: post #11

Member of the SyncFree Consortium (and committer to Antidote) here. Here's a video from J On The Beach this year on the details around the Just-Right Consistency approach that might help answer some questions: https://www.youtube.com/watch?v=Vd2I9v3pYpA

That video is really good - perfect for the layperson such as myself!

See also this video comparing Antidote to other kinds of data stores: https://youtu.be/oWUNCsFy-r0

Re: Antidote: CRDT-based distributed database

#84

Post by marc_shapiro »

runT1ME wrote: post #78
cmeiklejohn wrote: post #67

Earlier quoted context omitted.

(committer on Antidote, Riak and Lasp) We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.

CRDTs accumulate garbage, and need a global "sync" to GC. How do you mitigate this from having performance impacts?

This was true of the early CRDT designs, but they have improved since then. You will find plenty of inspiration here: http://dblp.uni-trier.de/pers/hd/b/Baquero:Carlos

Re: Antidote: CRDT-based distributed database

#85

Post by marc_shapiro »

kevmo314 wrote: post #6

This has that "too good to be true" vibe, and I can't find much information on the authors or the Syncfree Consortium organization that backs the project besides their own website. Is this at the cost of fast writes or flexible schema? The pitch video doesn't seem to mention any cons, yet seems to avoid mentioning the type of data or mutations supported. I guess I'll go read their publications.

A write is fast, because it happens directly at the closest replica, without any inter-replica synchronisation. There is no schema per se; rather the DB is object-oriented, and each application picks the object types it needs from the CRDT library. The library currently covers the basics: registers, flags, sets, maps, lists.

Re: Antidote: CRDT-based distributed database

#86

Post by marc_shapiro »

Horusiath wrote: post #72
teleclimber wrote: post #27

Earlier quoted context omitted.

> Making CRDTs consequence-free is ongoing research. What do you mean by that? I found possibly related language on this page the other day [0]: CRDTs are "[t]ypically not suited for editing application with consequent UI." Can you point me in the right direction? My googling got me nowhere. Thanks. [0] https://irisate.com/collaborative-editing-solutions-round-up...

It's not described in the docs, but elswhere [1] in their github repository you can find the note about rga , which is Replicated Growable Array CRDT, that can be used for building indexed linear sequences - shortly speaking arrays of elements with insert/remove semantics. [1]( https://github.com/SyncFree/antidote/blob/842874ca5ecffe947e... )

An efficient implementation of RGA designed for concurrent editing: http://dx.doi.org/10.1145/2957276.2957300

Re: Antidote: CRDT-based distributed database

#87

Post by marc_shapiro »

continuational wrote: post #30

How would I do a "put if absent" in this database? Is it efficient?

If the thing you "put" into is a CRDT, then two concurrent "put" will be merged, if that's OK for your application. If however you want to disallow concurrent "put"s then you need to add some concurrency control to the CRDT layer. See the CISE tool https://youtu.be/HJjWqNDh-GA, http://dx.doi.org/10.1145/2911151.2911160. Antidote doesn't yet support concurrency control, it's work in progress.

Re: Antidote: CRDT-based distributed database

#88

Post by marc_shapiro »

devit wrote: post #60
xinsight wrote: post #4

The video lists pros/cons for strongly consistent and eventually consistent databases, but only has pros for a "just right consistency" database. What are the cons?

Correct database implementations must be strongly consistent, unless your updates all commute with each other, in which case eventual consistency works. E.g. db[key] += value and db[key].insert(value) commute with themselves, but db[key] = value obviously doesn't. This just seems to be an attempt to implement a correct eventually consistent database, and the CRDTs are simply datatypes with commutative update operatio…

Some conditional updates are safe; others require to add concurrency control. Our CISE analyser will tell precisely you which side a specific operation falls into. See https://youtu.be/HJjWqNDh-GA and http://dx.doi.org/10.1145/2911151.2911160. Antidote doesn't yet support concurrency control, it's work in progress.

Re: Antidote: CRDT-based distributed database

#89

Post by runT1ME »

anne_biene wrote: post #81
runT1ME wrote: post #78

Earlier quoted context omitted.

CRDTs accumulate garbage, and need a global "sync" to GC. How do you mitigate this from having performance impacts?

[Disclaimer: I am an Antidote maintainer] Some CRDTs support garbage collection directly - if you run them in a causally consistent environment. Antidote is causally consistent and has a Set and Map implementation that work like this; for these CRDTs you don't need a global sync.

Thanks for the reply. Don't you negate some of the advantages of CRDTs by mandating casually consistent environments? Can you speak to that more?
Post reply on HN