Live data from Hacker News

OT and CRDT trade-offs for Real-Time collaboration

tiny.cloud

21–30 of 60 posts

Re: OT and CRDT trade-offs for Real-Time collaboration

#21

Wanderlog ( https://wanderlog.com ) is a Google Docs for planning travel, and naturally, we had to figure this out early in the process. If you're on a Node.js/React stack, we highly recommend using the combination of OT-JSON0 [1] and ShareDB [2], two excellent libraries. OT-JSON0 lets you perform operational transforms on any JSON-serializable structure pretty intuitively, and ShareDB handles synchronizing it betwee…

I came to the same "LWW on fragments" solution for peraspera.io. In a lot of cases, all OT gets you is being able to watch someone else's cursor move around. The new wore off on that a long time ago.

Re: OT and CRDT trade-offs for Real-Time collaboration

#22
post #20
post #18

Earlier quoted context omitted.

Really? > In OT, every user action is broken down into one or more operations. These operations are transmitted between clients along with their baseline reference; if two users perform actions at the same time, incoming operations must be transformed to include the local operations that have happened since that baseline. They are then applied locally and form the new baseline. > This constant transformation of opera…

But the article is arguing in favor of OT, not CRDT. A couple examples of those edge cases would go a long way, the one given is not very helpful to make a point.

What's wrong with the one given? If my cursor moves because some other user makes some other text bold, that doesn't sound like a desirable experience?

Re: OT and CRDT trade-offs for Real-Time collaboration

#23
post #19

Is the article suggesting to apply a string CRDT to an entire JSON structure? Are people doing that? At the very least, I would expect different JSON data types to use different CRDTs. For a collaborative model, I would expect the semantics of the data model to be reflected in the choice of CRDTs for different substructures in the JSON.

No, the state of the art CRDT solution for json is something like Automerge ( https://github.com/automerge/automerge ), that treats the document like a collection/combination of CRDTs of different types. (the author doesn't mention it explicitly, but it's linked at the end)

I have been using Automerge recently for a project, and I have found it to be very, very user-friendly.

Our use-case is offline-editing of documents with an eventual sync-with-yourself-online. It's mostly there as a sync tool, not as a p2p colalb editing. Unless you count yourself as a peer, I guess!

We store a document in local storage which is the result of `Automerge.save(automergedoc) => serializable string`. That can be loaded with `Automerge.load(s) => doc`.

"Changes" are a first-class concept in automerge (makes sense, I guess) so you can do

   before = Automerge.from({..some doc..})
   after = Automerge.change(before, "optional commit msg", (doc) => doc.hello = "world);
With that done, you can get a diff, `Automerge.getChanges(before, after) => [changes]`.

Those `[changes]` are what we try and send to a server and keep them in a `Set()` for each document (background sync, service worker, etc make a nice experience.

This is all wrapped up in Redux, and a middleware that captures the changes, and a reduxStore subscriber that puts the document back into local storage after any changes.

It is for the application programmer virtually transparent, and so far we've yet to find a fault with it.

I'm sure I'll come to regret making "changes" my principle data type, as the server implementation just sees serialized blobs of data, and has no concept of the structure of the document. That's solveable if I need it ever, because there's a protcol compatible implementation in Rust, and I could always use the JavaScript one on the server too, just haven't needed yet.

The serialization is also quite large, in the order of kilobytes for what are actually quite small docs, but they do contain the original commit messages too, and the Automerge team is planning to release an optimized serialization format early this year, which they say will be especially useful for fields with lots of changes, such as text fields, which currently pay a heavy toll in terms of tracking `[changes]`. Fortunately that also doesn't affect me.

I can also +1 some of the advice to keep your whole document in a CRDT. We use React+Redux, but since React Hooks made stateless functional components a reality, we felt it was appropriate to use React exclusively for the UI state, and track only document states in Redux reducers. It feels light-weight and sustainable, although we're only about a month and 5k LOC into the project. Time will tell I guess.

Re: OT and CRDT trade-offs for Real-Time collaboration

#24

w00h00, this is my area of expertise. After 8 years of working on this, I have changed my thoughts: - The correct algorithm is not always the correct user experience. - End-to-end encryption is too important to not have. - Offline support is great, but it behaving consistently is more important than it behaving "intently". - Biggest pain points can most easily be solved at the editing layer, not data layer. As a resu…

You are a charming and energized speaker and perhaps that is why Tim Draper thinks you a good investment as with others he has made. Acronyms like 'PTSD' and 'PARTY' in the project is likely good at attracting people who think they sound fun and do not question many details.

It is difficult to get brief clear details about how gun.js works. There are distracting or simple documents, discussions about high level ideas and tangentially-related topics. In some cases 'worse is better' yes but it sets a tone that the project is hiding internal problems of implementation or can not easy explain its own design.

The transparency of gun.js enterprise organizations also raises questions. Are people on your team related to you? Do you pay yourself and team and open source contributors?

Re: OT and CRDT trade-offs for Real-Time collaboration

#25

w00h00, this is my area of expertise. After 8 years of working on this, I have changed my thoughts: - The correct algorithm is not always the correct user experience. - End-to-end encryption is too important to not have. - Offline support is great, but it behaving consistently is more important than it behaving "intently". - Biggest pain points can most easily be solved at the editing layer, not data layer. As a resu…

With end-to-end encryption in GUN, what role does the server play? I assume the merging type activity happens in the clients?

BTW, the name makes it almost impossible to search for info about GUN.

Re: OT and CRDT trade-offs for Real-Time collaboration

#27

w00h00, this is my area of expertise. After 8 years of working on this, I have changed my thoughts: - The correct algorithm is not always the correct user experience. - End-to-end encryption is too important to not have. - Offline support is great, but it behaving consistently is more important than it behaving "intently". - Biggest pain points can most easily be solved at the editing layer, not data layer. As a resu…

Do you have any document that explain what resolution algorithm uses in what cases?

For example, one peer change a property value and the other peer deletes it.

Re: OT and CRDT trade-offs for Real-Time collaboration

#28
post #19

Earlier quoted context omitted.

No, the state of the art CRDT solution for json is something like Automerge ( https://github.com/automerge/automerge ), that treats the document like a collection/combination of CRDTs of different types. (the author doesn't mention it explicitly, but it's linked at the end)

I have been using Automerge recently for a project, and I have found it to be very, very user-friendly. Our use-case is offline-editing of documents with an eventual sync-with-yourself-online. It's mostly there as a sync tool, not as a p2p colalb editing. Unless you count yourself as a peer, I guess! We store a document in local storage which is the result of `Automerge.save(automergedoc) => serializable string`. Tha…

Yeah, personally the feature I'm looking forward to is materializing "changes" into a document without losing editing history.

At some point your set of changes grows too large or you want to trim it to, say, a few weeks or months. Afaik that isn't possible yet without losing the ability to merging later on.

Re: OT and CRDT trade-offs for Real-Time collaboration

#29
post #22
post #20

Earlier quoted context omitted.

But the article is arguing in favor of OT, not CRDT. A couple examples of those edge cases would go a long way, the one given is not very helpful to make a point.

What's wrong with the one given? If my cursor moves because some other user makes some other text bold, that doesn't sound like a desirable experience?

Because it doesn’t highlight a particularly relevant deficiency of the CRDT protocol. Preserving cursor position is not hard in that case. The post even mentions stronger downsides of OT, the example does not support the conclusion. Why is OT better?

Re: OT and CRDT trade-offs for Real-Time collaboration

#30
post #29
post #22

Earlier quoted context omitted.

What's wrong with the one given? If my cursor moves because some other user makes some other text bold, that doesn't sound like a desirable experience?

Because it doesn’t highlight a particularly relevant deficiency of the CRDT protocol. Preserving cursor position is not hard in that case. The post even mentions stronger downsides of OT, the example does not support the conclusion. Why is OT better?

Huh, the article gave me the impression that preserving the position was a crucial problem there.

As for the last question, I think the later posts will get to that.

Post reply on HN