11–20 of 60 posts
this is a wonderful writeup! thank you! I'm researching the same thing. Any tutorial on OT implementation? Especially, I need to handle structured data, like json.
The second post will cover which editor model we chose, and talk a little bit about the OT implementation details ;)
If you're on a Node.js/React stack, we highly recommend using the combination of OT-JSON0 [1] and ShareDB [2], two excellent libraries. OT-JSON0 lets you perform operational transforms on any JSON-serializable structure pretty intuitively, and ShareDB handles synchronizing it between multiple clients and resolving conflicts. Building your own is going to be a pain, so unless your core business is a text editor like TinyMCE, it's probably best to stand on the shoulders of giants.
Another approach that we didn't end up using is Figma's: especially if you're working on anything other than text editing, ask if you actually want to use operational transform. Something simpler might be good enough -- in Figma's case, they use "last write wins" because they were able to break the document into small enough, atomic elements. [3]
[1] https://www.npmjs.com/package/ot-json0
[2] https://www.npmjs.com/package/sharedb
[3] https://www.figma.com/blog/how-figmas-multiplayer-technology...
this is a wonderful writeup! thank you! I'm researching the same thing. Any tutorial on OT implementation? Especially, I need to handle structured data, like json.
I’ve been doing a pretty deep dive on CRDTs and OTs to get the right user experience for our collaborative website builder[1]. I’m still not done with the implementation and subsequent testing (i.e., splitting text nodes, as mentioned in the post). But the core algorithm behind Automerge[2], formalized in the OpSets paper[3] is extremely promising. A good example of its power is the ability to do an atomic tree “move…
At the very least, I would expect different JSON data types to use different CRDTs. For a collaborative model, I would expect the semantics of the data model to be reflected in the choice of CRDTs for different substructures in the JSON.
After 8 years of working on this, I have changed my thoughts:
- The correct algorithm is not always the correct user experience.
- End-to-end encryption is too important to not have.
- Offline support is great, but it behaving consistently is more important than it behaving "intently".
- Biggest pain points can most easily be solved at the editing layer, not data layer.
As a result, OT gets thrown out the door immediately.
I built the most popular CRDT powered database on the market (https://github.com/amark/gun, 11K+ stars, 11M+ monthly installs), but have some harsh words for CRDT obsessed people:
CRDTs are great, but if you create an infinitely complex one to handle "every word editing operation" it'll actually result in an inferior user experience.
As such, for example, GUN supports lock-free concurrent operations on any depth of data in a graph, but keeping text/strings behavior atomic is way more important than having built-in automatic string merges.
Why? It is much simpler to build a correct text/string CRDT on top of a predictable, stable, graph CRDT that is understandable.
Another example is, in our blog editing tools, despite having spent years researching & implementing (+ others in our community) precise character-by-character CRDT resolution schemes, I found I personally had a much better offline & local first user experience with a predictable per-paragraph sync scheme.
This is a stupid simple approach, but what it means is that I as a user, can easily predict whats going to happen, so if I'm offline I know I should just copy a new paragraph and fiddle with things there, cause it isn't gonna get "auto delete regrammar merged".
Generally speaking, me and colleagues don't "fight" over the same paragraph, we're usually concurrently writing different "sections" at the same time, it is pretty rare to grammar fix their edit same paragraph as they're typing it - that is also just kinda, rude looking.
Anyways, finally, is that the majority of text styling and DOM edge cases can be handled by having a deterministic canonical DOM hierarchy that always gets applied at the editing layer, BEFORE any CRDT operations even occur.
So for instance, re-arrange the DOM tree such that is always inside inside , or something like that.
We built this into a library called normalize, and it instantly eliminated so much complexity, especially at the CRDT level. Ping me if you want a demo of this library.
Finally, for everyone else, we also built an INTERACTIVE CARTOON EXPLAINER of an extremely basic text CRDT to help others understand the more detailed concepts:
So.. we’re left with no real explanation to why OT is better than CRDT other than a 5s video showing a poorly managed text node update that moves the cursor. I feel tricked.
> In OT, every user action is broken down into one or more operations. These operations are transmitted between clients along with their baseline reference; if two users perform actions at the same time, incoming operations must be transformed to include the local operations that have happened since that baseline. They are then applied locally and form the new baseline.
> This constant transformation of operations turned out to have too many edge cases where clients were found to not produce the same baseline (the "wrong" papers above). When that happens, the clients will never converge on the same result and break the fundamental assumption of collaboration.
And that’s exactly right. It’s fairly easy to have an OT system that is very vulnerable, because clients can cheaply generate change sets that are extremely computationally expensive on the server side. I’ve seen a system where a single mobile phone could, in a few seconds, lock up the synchronisation server for days.
Is the article suggesting to apply a string CRDT to an entire JSON structure? Are people doing that? At the very least, I would expect different JSON data types to use different CRDTs. For a collaborative model, I would expect the semantics of the data model to be reflected in the choice of CRDTs for different substructures in the JSON.
So.. we’re left with no real explanation to why OT is better than CRDT other than a 5s video showing a poorly managed text node update that moves the cursor. I feel tricked.
Really? > In OT, every user action is broken down into one or more operations. These operations are transmitted between clients along with their baseline reference; if two users perform actions at the same time, incoming operations must be transformed to include the local operations that have happened since that baseline. They are then applied locally and form the new baseline. > This constant transformation of opera…