21–30 of 49 posts
The death stroke for these types of projects seems to be lack of funding. This project is sponsored by nlnet[0] providing between 5k - 50k EU per year. Let's hope this gets additional resources. As a note, it appears to use Elastic's 2.0 license preventing selling software that includes this library [1] [0] https://nlnet.nl/project/JSON-Joy/ [1] https://github.com/streamich/json-joy/blob/master/LICENSE
Earlier quoted context omitted.
General CRDTs will guarantee valid data structures, but not schema/domain model validity. Kind of like how CRDTs applied to text will guarantee a string, but not valid English.
This is a great analogy for something I've struggled to put into words. I’ll add: if you have invariants, you almost by definition have conflicts. The C in CRDT is for conflict-free, so if you can have conflicts in the data domain you probably want something that can preserve them (like state machine synchronization) rather than a CRDT.
We can do this today with keys / values pretty easily with “MV (multi value) Registers”. If two writers concurrently set the value, the next read can see both values and decide what to do about that.
But nobody (as far as I know) has yet made a crdt for code editing which uses this same trick to mark & annotate editing conflicts. And that seems like a missing piece if we want to replace git with something better.
What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
You need to design your document to minimize these kinds of issues. You can treat properties in the document as a last-write-wins register to minimize “strange merge” consistency within that property. For use cases with a central server, you can use server reconciliation to handle “true conflicts”, essentially doing layer of OT-style ops at the application level around CRDT structures provided by a library. See how R…
What’s the future of this work? Is it planned to be entirely standalone? Planning to build integration with popular editors? Planning a websocket server implementation with hooks etc / a hosted solution?
Subscribe on Substack for updates.
What I have not understood yet is how do you preserve invariants over a merge of JSON crdt. What do you do when a document has a structure that can be represented as a json, but not every valid json is a valid document? How do you avoid merges producing valid jsons but invalid documents?
General CRDTs will guarantee valid data structures, but not schema/domain model validity. Kind of like how CRDTs applied to text will guarantee a string, but not valid English.
Very interested in seeing progress, JSON CRDTs benchmarked with JS objects and arrays, I currently have an implementation based on Automerge but it's borderline nonviable on our largest data structures.
I really want json based CRDTs to be fast, but one of the problems we have optimising this stuff is that there aren’t a lot of real world data traces around to use as baselines for benchmarking. I don’t know which parts of automerge are slow, and without that knowledge I can’t make them fast. If we have some data from your application, most upcoming json based CRDTs will almost certainly work well for your use case.
If you’re up for it, flick me an email or just open a PR on https://github.com/josephg/editing-traces
I'm still hoping for a CRDT implementation with robust, thoroughly tested libraries for both Python and JavaScript that can talk to each other - I want to run Python on the server and JavaScript in the client and keep the two in sync with each other. Closest I've seen to that is automerge but the Python version doesn't appear to be actively maintained or packaged for PyPI yet: https://github.com/automerge/automerge-p…
Wouldn't any of the Rust implementations be what you need (Yrs, Diamond Types or the Automerge rust re-write)? Given you can bind to a Rust implementation in JS or Python or whatever else.
More importantly, I prefer not to be the only user of something like this - a Rust-Python wrapper that someone else has already tried running in production is a lot more trustworthy than something I knock together myself.
I'm still hoping for a CRDT implementation with robust, thoroughly tested libraries for both Python and JavaScript that can talk to each other - I want to run Python on the server and JavaScript in the client and keep the two in sync with each other. Closest I've seen to that is automerge but the Python version doesn't appear to be actively maintained or packaged for PyPI yet: https://github.com/automerge/automerge-p…
Wouldn't any of the Rust implementations be what you need (Yrs, Diamond Types or the Automerge rust re-write)? Given you can bind to a Rust implementation in JS or Python or whatever else.
Earlier quoted context omitted.
Wouldn't any of the Rust implementations be what you need (Yrs, Diamond Types or the Automerge rust re-write)? Given you can bind to a Rust implementation in JS or Python or whatever else.
I want someone else to write the Rust-to-Python bindings for me because I'm lazy! More importantly, I prefer not to be the only user of something like this - a Rust-Python wrapper that someone else has already tried running in production is a lot more trustworthy than something I knock together myself.