Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antidote: CRDT-based distributed database

syncfree.github.io

Post reply on HN
Print view View original

61–70 of 89 posts

Re: Antidote: CRDT-based distributed database

#61

Post by takeda »

cmeiklejohn wrote: post #38
davidw wrote: post #35

I wonder what it looks like in terms of resource usage. I think there's a strong case for something like this for IoT type devices. Imagine the simple case of adding a name to a contact list on a mobile device, and wanting that to get synced up with a series of other devices.

(committer on Antidote, author of Lasp, both SyncFree projects) Antidote is designed for DC deployments, in the hundreds of clusters. It provides causal consistency and transactions. Lasp is designed for high-churn, edge deployments (with CRDTs) and provides eventual consistency: it's designed for 1,000+ nodes. They share a bit of code, and you might want to evaluate both to determine which is a better use case: Lasp…

This is based on riak which is written in erlang.

Those nodes connect in a mesh, and actually the limit of scaling is about 100 nodes in one cluster.

Re: Antidote: CRDT-based distributed database

#62

Post by [deleted] »

dang wrote: post #48
marknadal wrote: post #16

Hey, author of GUN (the current most popular generalizable CRDT based database), and want to say I'm impressed. I'm often the first to nitpick things but this looks great: - Built in Erlang - Great explainer videos - Well documented CRDTs that you accept - Team of university related researches in CRDTs. I'll be looking through your guys stuff more. But good job! We need more people like you guys out there.

We detached this comment from https://news.ycombinator.com/item?id=15863587 and marked it off-topic.

[deleted]

Re: Antidote: CRDT-based distributed database

#63

Post by cmeiklejohn »

takeda wrote: post #61
cmeiklejohn wrote: post #38

Earlier quoted context omitted.

(committer on Antidote, author of Lasp, both SyncFree projects) Antidote is designed for DC deployments, in the hundreds of clusters. It provides causal consistency and transactions. Lasp is designed for high-churn, edge deployments (with CRDTs) and provides eventual consistency: it's designed for 1,000+ nodes. They share a bit of code, and you might want to evaluate both to determine which is a better use case: Lasp…

This is based on riak which is written in erlang. Those nodes connect in a mesh, and actually the limit of scaling is about 100 nodes in one cluster.

See my comment above that you specifically replied to.

Lasp [in high-scalability mode] does not connect in a mesh, and the links I provided demonstrate that we do not connect in a mesh.

I specifically provided a link in the comment [with a link to an academic publication and a paper] you responded to about how we do non-full mesh and achieve 1000+ nodes.

Antidote only uses distributed Erlang inside of a single cluster limiting a single cluster to 100+ nodes. The multi-DC protocol is not based on distributed Erlang, and therefore, is not limited by distributed Erlang (but, is limited by other factors.)

Additionally, there are Riak Core modifications (authored by myself and others) that bypass distributed Erlang using the same semantics, and therefore, don't have the 100+ node cluster restriction.

And finally, the 100+ node cluster restriction is highly-debated, by myself and other academics, because the restrictions are based on use of the global module, where systems that don't use this module will experience higher scalability numbers (based on empirical evidence, not yet published.) For reference, Ericsson, has run 200+ node Erlang clusters.

Re: Antidote: CRDT-based distributed database

#65

Post by bpicolo »

KirinDave wrote: post #39
teleclimber wrote: post #27

Earlier quoted context omitted.

> Making CRDTs consequence-free is ongoing research. What do you mean by that? I found possibly related language on this page the other day [0]: CRDTs are "[t]ypically not suited for editing application with consequent UI." Can you point me in the right direction? My googling got me nowhere. Thanks. [0] https://irisate.com/collaborative-editing-solutions-round-up...

I dunno about that quote, Treedoc works pretty well. In general there are tradeoffs to CRDTs and not everyone loves those tradeoffs. As time goes on we find CRDTs that make better tradeoffs. For example, the earliest examples of CRDT sets grew linearly with the values added to the set, without regard for deletion. For all time. That's a pretty steep cost.

LSEQ appears to improve upon treedoc in ways, too: https://hal.archives-ouvertes.fr/hal-00921633/document

Re: Antidote: CRDT-based distributed database

#66

Post by sturadnidge »

cmeiklejohn wrote: post #11

Member of the SyncFree Consortium (and committer to Antidote) here. Here's a video from J On The Beach this year on the details around the Just-Right Consistency approach that might help answer some questions: https://www.youtube.com/watch?v=Vd2I9v3pYpA

That video is really good - perfect for the layperson such as myself!

Re: Antidote: CRDT-based distributed database

#67

Post by cmeiklejohn »

skrebbel wrote: post #5

At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…

(committer on Antidote, Riak and Lasp)

We're investigating a backend that works on LevelDB and RocksDB. We just haven't had the academic resources to get it implemented yet. However, it's largely an engineering resource problem and not a theoretical problem.

Re: Antidote: CRDT-based distributed database

#68

Post by jlu »

shalabhc wrote: post #51

Related ideas exist in David Reed's 'Atomic Actions' which use pseudotime: http://www.cs.sfu.ca/~vaughan/teaching/431/papers/reed83.pdf : "thinking about objects as sequences of unchangeable versions, object histories" "the correct construction and execution of a new atomic action can be accomplished without knowledge of all other atomic actions in the system that might execute concurrently" Is anyone aware of compar…

FYI, that paper was from 1983, CRDT was introduced 2007, conceptually they share the same idea, but most modern CRDT implementations tends to use Lamport timestamp to causally solve conflicts.

In short, antidote looks like a decent solution to this kind of problems.

Re: Antidote: CRDT-based distributed database

#69

Post by jlu »

fiatjaf wrote: post #32

I wanted a database that would receive "events" asynchronously and stored that, but at the same time would process these events (from some piece of previously written code) to generate a queryable schema. If I wanted to change the schema later, the database would let me just rewrite the code and it would reprocess all the received events since the beggining. My use case is not anything high-performance or with thousa…

Sounds like kappa architecture is something that will fit your request.

http://milinda.pathirage.org/kappa-architecture.com/

Post reply on HN