What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
CRDT Benchmarks
11–20 of 49 posts
Re: CRDT Benchmarks
#12I'm still hoping for a CRDT implementation with robust, thoroughly tested libraries for both Python and JavaScript that can talk to each other - I want to run Python on the server and JavaScript in the client and keep the two in sync with each other. Closest I've seen to that is automerge but the Python version doesn't appear to be actively maintained or packaged for PyPI yet: https://github.com/automerge/automerge-p…
That ypy library looks very interesting. Do you have any idea what considerations would need to be kept in mind when trying to implement CRDTs?
I'm thinking about something relatively basic, let's say a shared markdown document and/or shared dataframe view of a database table?
Re: CRDT Benchmarks
#13What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
General CRDTs will guarantee valid data structures, but not schema/domain model validity. Kind of like how CRDTs applied to text will guarantee a string, but not valid English.
Re: CRDT Benchmarks
#14What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
General CRDTs will guarantee valid data structures, but not schema/domain model validity. Kind of like how CRDTs applied to text will guarantee a string, but not valid English.
This is a great analogy for something I've struggled to put into words.
I’ll add: if you have invariants, you almost by definition have conflicts. The C in CRDT is for conflict-free, so if you can have conflicts in the data domain you probably want something that can preserve them (like state machine synchronization) rather than a CRDT.
Re: CRDT Benchmarks
#15What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
You need to design your document to minimize these kinds of issues. You can treat properties in the document as a last-write-wins register to minimize “strange merge” consistency within that property.
For use cases with a central server, you can use server reconciliation to handle “true conflicts”, essentially doing layer of OT-style ops at the application level around CRDT structures provided by a library. See how Replicache suggests handling text for example. They provide a server reconciliation framework, and suggest you use a CRDT for text within that semantics.
Re: CRDT Benchmarks
#16Great to see this comparison! I haven't heard about Json-joy yet, so I'm curious to learn more. We are using Yjs in production and it works like magic!
Re: CRDT Benchmarks
#17I need to dig into this more, but I'm sceptical of only benchmarking ops/second, that's not really a problem that needs solving, the existing toolkits are fast enough. Also, this benchmark doesn't show document size and growth, that is something where more research is needed. Always excited for any CRDT innovations though, and I'm sure there is stuff to learn from this work.
Not always fast enough. Automerge with 32MB of JSON to parse is... painfully slow.
Re: CRDT Benchmarks
#18The death stroke for these types of projects seems to be lack of funding. This project is sponsored by nlnet[0] providing between 5k - 50k EU per year. Let's hope this gets additional resources. As a note, it appears to use Elastic's 2.0 license preventing selling software that includes this library [1] [0] https://nlnet.nl/project/JSON-Joy/ [1] https://github.com/streamich/json-joy/blob/master/LICENSE
Apache 2.0 as of... 18min ago?
Re: CRDT Benchmarks
#19The death stroke for these types of projects seems to be lack of funding. This project is sponsored by nlnet[0] providing between 5k - 50k EU per year. Let's hope this gets additional resources. As a note, it appears to use Elastic's 2.0 license preventing selling software that includes this library [1] [0] https://nlnet.nl/project/JSON-Joy/ [1] https://github.com/streamich/json-joy/blob/master/LICENSE
[1] is a bummer. Turns this project into a technology showcase without any practical use.
Re: CRDT Benchmarks
#20I'm still hoping for a CRDT implementation with robust, thoroughly tested libraries for both Python and JavaScript that can talk to each other - I want to run Python on the server and JavaScript in the client and keep the two in sync with each other. Closest I've seen to that is automerge but the Python version doesn't appear to be actively maintained or packaged for PyPI yet: https://github.com/automerge/automerge-p…
Wouldn't any of the Rust implementations be what you need (Yrs, Diamond Types or the Automerge rust re-write)? Given you can bind to a Rust implementation in JS or Python or whatever else.