Live data from Hacker News

Collaborative text editing with Eg-Walker: Better, faster, smaller

arxiv.org

21–30 of 32 posts

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#21
post #14
post #12

Earlier quoted context omitted.

This seems to be a holy grail, to be honest! Super-simple database representations with barely any processing required on the "write path," instant startup, minimal memory requirements on both server and client without a need for CRDT data structures to be in memory, none of the O(n^2) complexity of OT. In fact, if I'm interpreting it correctly, it should be straightforward to get this working in a serverless environ…

Author here. Thanks! Yeah this is my hope too. Egwalker has one other advantage here: the data format will be stable and consistent. With CRDTs, every different crdt algorithm (Yjs, automerge/rga, fugue, etc) actually stores different fields on disk. So if someone figure out a new way to make text editing work better, we need to rip up our file formats and network protocols. Egwalker just stores the editing events in…

I’ve loved learning all your detailed info on CRDT work. Thank you for progressing the field!

Since it stores all the editing events, does this mean that the complexity of opening a document is at least O(N) in terms of number of edits? Or are there interim snapshots / merging / and/or intelligent range computations to reduce the number of edits that need to be processed?

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#22
post #14

Earlier quoted context omitted.

Author here. Thanks! Yeah this is my hope too. Egwalker has one other advantage here: the data format will be stable and consistent. With CRDTs, every different crdt algorithm (Yjs, automerge/rga, fugue, etc) actually stores different fields on disk. So if someone figure out a new way to make text editing work better, we need to rip up our file formats and network protocols. Egwalker just stores the editing events in…

I’ve loved learning all your detailed info on CRDT work. Thank you for progressing the field! Since it stores all the editing events, does this mean that the complexity of opening a document is at least O(N) in terms of number of edits? Or are there interim snapshots / merging / and/or intelligent range computations to reduce the number of edits that need to be processed?

You can just store a snapshot on disk (ie, the raw text) and load that directly. You only ever need to look at historical edits when merging concurrent changes into the local document state. (And thats pretty rare in practice).

Even when that happens, the algorithm only needs to look at operations as far back as the most recent "fork point" between the two branches in order to merge. (And we can compute that fork point in O(n) time - where n is the number of events that have happened since then). Its usually very very fast.

In an application like google docs or a wiki, the browser will usually never need to look at any historical changes at all in order to edit a document.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#23
post #17

Earlier quoted context omitted.

First paragraph: yes, exactly. > OTs take an event.. This is how the early Jupiter OT works, yes. And most OT systems work like this. But there are also some papers on more recent OT systems which can work with more than 2 peers. Unfortunately, many of these systems have turned out to have convergence bugs and/or they are O(n^2). For our paper one of our example datasets takes tens of milliseconds to replay with CRDT…

This is probably a question about classic CRDTs as much as eg-walker: Do all possible topological sorts of the event graph result in the same final consensus document? If yes how do we know that, and if no, how do they resolve the order in which each branch is applied?

> Do all possible topological sorts of the event graph result in the same final consensus document?

Yes. Thats usually referred to as the "convergence property".

> If yes how do we know that

Usually, careful design, mathematical proofs and randomized (fuzz) testing. Fuzz testing is absolutely essential - In over a decade of working on systems like this, I don't know if I've ever implemented something correctly first try. Fuzz testing is essential. You shouldn't trust the correctness of any system which haven't been sufficiently fuzzed. (Luckily, fuzzers are easy to write, and the convergence property is very easy to test for.)

For Eg-walker, I think we've pumped around 100M randomly generated events (in horribly complex graphs) through our implementation to flush out any bugs.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#24
post #23

Earlier quoted context omitted.

This is probably a question about classic CRDTs as much as eg-walker: Do all possible topological sorts of the event graph result in the same final consensus document? If yes how do we know that, and if no, how do they resolve the order in which each branch is applied?

> Do all possible topological sorts of the event graph result in the same final consensus document? Yes. Thats usually referred to as the "convergence property". > If yes how do we know that Usually, careful design, mathematical proofs and randomized (fuzz) testing. Fuzz testing is absolutely essential - In over a decade of working on systems like this, I don't know if I've ever implemented something correctly first…

This seems to be a field perfect for theorem proving, I think I've seen some work by Kleppmann using Isabelle.

I once tried to understand the Yjs paper, but I came to the conclusion that their proof is just wrong! They do some impressively looking logical reasoning in the paper, but they define some order in terms of itself, so they don't really show anything, if I remember correctly. If you tried that in Isabelle, it would stop you already at the very start of all that nonsense.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#25
post #23

Earlier quoted context omitted.

> Do all possible topological sorts of the event graph result in the same final consensus document? Yes. Thats usually referred to as the "convergence property". > If yes how do we know that Usually, careful design, mathematical proofs and randomized (fuzz) testing. Fuzz testing is absolutely essential - In over a decade of working on systems like this, I don't know if I've ever implemented something correctly first…

This seems to be a field perfect for theorem proving, I think I've seen some work by Kleppmann using Isabelle. I once tried to understand the Yjs paper, but I came to the conclusion that their proof is just wrong! They do some impressively looking logical reasoning in the paper, but they define some order in terms of itself, so they don't really show anything, if I remember correctly. If you tried that in Isabelle, i…

I talked to Kevin Jahns (the author of the YATA paper & Yjs) about his paper a few years ago. He said he found errors in the algorithm described in the paper, after it was published. The algorithm he uses in Yjs is subtly different from YATA in order to fix the mistakes.

He was quite surprised the mistakes went unnoticed through the peer review process.

There have also been some (quite infamous) OT algorithm papers which contain proofs of correctness, but which later turned out to actually be incorrect. (Ie, the algorithms don't actually converge in some instances).

I'm embarassed to say I don't know Isabelle well enough to know how you would use it to prove convergence properties. But I have gotten very good at fuzz testing over the years. Its wild how many bugs in seemingly-working software I've found using the technique.

I think ideally you'd use both approaches.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#26
post #22

Earlier quoted context omitted.

I’ve loved learning all your detailed info on CRDT work. Thank you for progressing the field! Since it stores all the editing events, does this mean that the complexity of opening a document is at least O(N) in terms of number of edits? Or are there interim snapshots / merging / and/or intelligent range computations to reduce the number of edits that need to be processed?

You can just store a snapshot on disk (ie, the raw text) and load that directly. You only ever need to look at historical edits when merging concurrent changes into the local document state. (And thats pretty rare in practice). Even when that happens, the algorithm only needs to look at operations as far back as the most recent "fork point" between the two branches in order to merge. (And we can compute that fork poi…

Very clever idea. Thanks for explaining

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#27
post #25

Earlier quoted context omitted.

This seems to be a field perfect for theorem proving, I think I've seen some work by Kleppmann using Isabelle. I once tried to understand the Yjs paper, but I came to the conclusion that their proof is just wrong! They do some impressively looking logical reasoning in the paper, but they define some order in terms of itself, so they don't really show anything, if I remember correctly. If you tried that in Isabelle, i…

I talked to Kevin Jahns (the author of the YATA paper & Yjs) about his paper a few years ago. He said he found errors in the algorithm described in the paper, after it was published. The algorithm he uses in Yjs is subtly different from YATA in order to fix the mistakes. He was quite surprised the mistakes went unnoticed through the peer review process. There have also been some (quite infamous) OT algorithm papers w…

Ah, that makes sense! I thought that Yjs must be doing something differently than described, because it seems to work well in practice, but I couldn't see how Yata would. Anyway, I learnt a lot by thinking through that paper :-)

Fuzz testing and proof are complementary, I think, both catch things the other one might not have caught. The advantage of Fuzz testing is that it tests the real thing, not a mathematical replica of it.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#28

Do collaborative whiteboard like software use the same algorithms, or are there more suitable algorithms for picture collaborations?

They usually use a central server and last-writer-wins semantics.

Figma for example https://www.figma.com/blog/how-figmas-multiplayer-technology...

I've seen CF Durable Objects used quite a lot.

There are other emerging patterns too: https://www.instantdb.com/

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#29
post #13

Earlier quoted context omitted.

I find the formulation in the abstract slightly confusing. As far as I understand EG-Walker is a CRDT, an operation-based one.

Author here. It’s kinda both a crdt and an operational transform system. It’s a crdt in that all peers share & replicate the set of all editing events. (A grow-only set crdt if we’re being precise). Peers can use those editing events to generate the document state at any point in time, merge changes and so on. But the editing events themselves are stored and expressed in their “original” form (unlike existing CRDTs,…

Ok, I started reading the paper now, and this seems to be a really cool method. I didn't understand all the details of apply/retreat/advance yet, though.

I am wondering, the EG graph is a very general construct, and the events themselves (Insert(i, c) and Delete(i)) are very natural as well. You say in the paper this should also work for other applications than plain text, but I guess then another CRDT has to be constructed to implement apply/retreat/advance. Would it be possible to formulate all of this independently of the application and particular CRDT, together with corresponding correctness theorems? That would help with constructing versions of this for other applications, and maybe make understanding this particular application for plain text easier.

Re: Collaborative text editing with Eg-Walker: Better, faster, smaller

#30
post #13

Earlier quoted context omitted.

Author here. It’s kinda both a crdt and an operational transform system. It’s a crdt in that all peers share & replicate the set of all editing events. (A grow-only set crdt if we’re being precise). Peers can use those editing events to generate the document state at any point in time, merge changes and so on. But the editing events themselves are stored and expressed in their “original” form (unlike existing CRDTs,…

Ok, I started reading the paper now, and this seems to be a really cool method. I didn't understand all the details of apply/retreat/advance yet, though. I am wondering, the EG graph is a very general construct, and the events themselves (Insert(i, c) and Delete(i)) are very natural as well. You say in the paper this should also work for other applications than plain text, but I guess then another CRDT has to be cons…

Maybe. Here's another way to think of the algorithm:

All the complexity comes about because we're trying to convert the insert / delete position from edits (expressed at their original version) to some later current version.

There's lots of ways of solving this problem. For example, we could build a data structure which contains metadata for every inserted item in a text document. For every inserted character, we store when the item was inserted and when (if ever) the item was deleted.

Then you could implement the algorithm in a simpler way. Lets say I'm trying to insert at position 1000, at some version V.

- We scan the list of characters from the start of the document, looking for the 1000th item which was actually in the document at version V.

- For each character in the list, we can tell if that item was inserted at version V by comparing V to the stored inserted / deleted at times.

This algorithm would be correct, and it avoids retreat / advance. The only problem with this approach is that it would be slow - because you're constantly scanning the document to convert insert positions. Inserting N items into a document take O(N^2) time.

The retreat / advance approach described in the paper is an optimization on top of this algorithm which performs the same work in O(N log N) time.

I wish we made this more clear in the paper. In an earlier draft we spent about 5 pages simply talking about version theory. The algorithm was then described using that theory with a stronger theoretical grounding. But I think that description may have been even more confusing.

> You say in the paper this should also work for other applications than plain text, but I guess then another CRDT has to be constructed to implement apply/retreat/advance. Would it be possible to formulate all of this independently of the application and particular CRDT, together with corresponding correctness theorems?

"Independently of the application and particular CRDT"? I don't know, we might have to think through how that would work for every CRDT. Do you have any personal favorites that would be worth thinking through?

For registers (eg in a variable, dictionary, hash map or array where indexes never change), you could implement a similar algorithm incredibly easily by just doing the version comparison operation on the graph. (The current value is the value set in the graph's frontier.) The retreat / advance optimisation isn't needed at all for registers.

For a list - for example, a list of layers in photoshop - we might need something more complex, since layers can be inserted / deleted like text and as a result the index of subsequent items changes. But layers can also be reordered - and that requires some thought. For rich text, there's an approach that I think would work but I haven't implemented it yet.

Post reply on HN