Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Antidote: CRDT-based distributed database

syncfree.github.io

Post reply on HN
Print view View original

21–30 of 89 posts

Re: Antidote: CRDT-based distributed database

#21

Post by lopatin »

gjem97 wrote: post #7
kevmo314 wrote: post #6

This has that "too good to be true" vibe, and I can't find much information on the authors or the Syncfree Consortium organization that backs the project besides their own website. Is this at the cost of fast writes or flexible schema? The pitch video doesn't seem to mention any cons, yet seems to avoid mentioning the type of data or mutations supported. I guess I'll go read their publications.

I think the issue you're looking for is a limited number of supported datatypes. This doesn't appear to be a general-purpose database at the moment: don't expect to drop it in as a replacement for MySQL. Doesn't mean it can't be in the future, but it would have to fall back on other techniques for the datatypes that can't be built with CRDTs. http://syncfree.github.io/antidote/crdts.html

I think the data types look fine. If I'm using such a database, I don't think I want the CRDT data types to be abstracted away behind some general-purpose (SQL?) interface. Can that even be done? I'd rather just pay the small price of restructuring my data into maps and sets and have the CRDT database do the rest without any extra indirection.

Re: Antidote: CRDT-based distributed database

#22

Post by cmeiklejohn »

cmeiklejohn wrote: post #13
cmeiklejohn wrote: post #11

Member of the SyncFree Consortium (and committer to Antidote) here. Here's a video from J On The Beach this year on the details around the Just-Right Consistency approach that might help answer some questions: https://www.youtube.com/watch?v=Vd2I9v3pYpA

Here's links to other things in the SyncFree group. Lasp (previously on HN): https://news.ycombinator.com/item?id=14300763 https://news.ycombinator.com/item?id=15444156 SyncFree project review results: http://cordis.europa.eu/result/rcn/197514_en.html Legion: https://syncfree.lip6.fr/index.php/2-uncategorised/54-legion SyncFree homepage: http://syncfree.lip6.fr

Further follow-up: we're (members of the SyncFree consortium and it's follow up project for moving CRDTs, Antidote, and Lasp to the edge) considering raising money to industrialize Antidote (and it's associated tool chain) for building Just-Right Consistency applications.

If you'd be interested in any of the above, or just using the database and/or trying the tools, reach out to me via email christopher.meiklejohn at gmail.com or cmeik on Twitter.

Re: Antidote: CRDT-based distributed database

#23

Post by nine_k »

dustingetz wrote: post #10
kevmo314 wrote: post #6

This has that "too good to be true" vibe, and I can't find much information on the authors or the Syncfree Consortium organization that backs the project besides their own website. Is this at the cost of fast writes or flexible schema? The pitch video doesn't seem to mention any cons, yet seems to avoid mentioning the type of data or mutations supported. I guess I'll go read their publications.

I don't think it encodes a graph that you query (like, say, Neo4J or Postgres with an orm); it seems to give various replicated data structure primitives, like documents, basically a kv store? So for building something like google docs.

One glaring omission is indexes.

You can likely build something SQL-like to form complex queries coordinating related objects, but the efficiency of such queries is not a given. You can likely manually build something like an index to speed up lookups using e.g. the map primitive. With a plan for a complex query, you're on your own.

Another likely problematic part is PK generation. You can of course use a counter, but I'm not sure how efficient will it be with burst inserts. For that, you'd have to come up with client-generated PKs that are unlikely to clash, e.g. UUIDs.

Re: Antidote: CRDT-based distributed database

#24

Post by nine_k »

skrebbel wrote: post #5

At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…

It appears to use Riak as the storage layer.

Re: Antidote: CRDT-based distributed database

#25

Post by nine_k »

KirinDave wrote: post #19
erik_seaberg wrote: post #17

Earlier quoted context omitted.

The biggest drawback is that only some operations can be supported. E.g., without strong consistency you can detect double-spending from an account but you can't prevent it, because the validity of an operation can't depend on operations a datacenter hasn't seen.

Financial examples are bad because in fact the financial world IS eventually consistent. It's quite possible to withdraw the same $100 from an account via multiple ATM machines.

With ATMs and debit cards, I thinks it's generally not true, they seem to use the online mode and update the balance of a checking account within seconds.

With credit cards, you can indeed start more transactions against the same balance, and you're never sure in which order they will complete.

Re: Antidote: CRDT-based distributed database

#26

Post by cmeiklejohn »

nine_k wrote: post #24
skrebbel wrote: post #5

At first glance, this looks amazing. I truly believe CRDTs are the solution to lots of distributed systems problems, and that exposing their characteristics to developers directly, rather than trying to abstract them away in nicer, but leaky, abstractions, is the right way to go. That said, a major part of why databases are hard is that reliable storage is hard. I see remarkably little about this on Antidote's homepa…

It appears to use Riak as the storage layer.

(committer here)

It uses Riak Core as the underlying distribution layer, which is the same that Riak itself uses. For storage of the log, the built-in Erlang disk_log is used.

Re: Antidote: CRDT-based distributed database

#27

Post by teleclimber »

KirinDave wrote: post #18
kevmo314 wrote: post #6

This has that "too good to be true" vibe, and I can't find much information on the authors or the Syncfree Consortium organization that backs the project besides their own website. Is this at the cost of fast writes or flexible schema? The pitch video doesn't seem to mention any cons, yet seems to avoid mentioning the type of data or mutations supported. I guess I'll go read their publications.

It's a government sponsored project so one would expect their publicity to be limited. That's part of the reason I posted it. Stuff like this is extremely exciting. It's at the cost of playing by the rules of CRDTs. Making CRDTs consequence-free is ongoing research.

> Making CRDTs consequence-free is ongoing research.

What do you mean by that?

I found possibly related language on this page the other day [0]:

CRDTs are "[t]ypically not suited for editing application with consequent UI."

Can you point me in the right direction? My googling got me nowhere. Thanks.

[0] https://irisate.com/collaborative-editing-solutions-round-up...

Re: Antidote: CRDT-based distributed database

#29

Post by erik_seaberg »

nine_k wrote: post #25
KirinDave wrote: post #19

Earlier quoted context omitted.

Financial examples are bad because in fact the financial world IS eventually consistent. It's quite possible to withdraw the same $100 from an account via multiple ATM machines.

With ATMs and debit cards, I thinks it's generally not true, they seem to use the online mode and update the balance of a checking account within seconds. With credit cards, you can indeed start more transactions against the same balance, and you're never sure in which order they will complete.

When a payment processing system doesn't promptly cancel auth holds, customers definitely complain about being prevented from spending their money. This stuff should be table stakes but some retail banks are just way behind.
Post reply on HN