You don't need a CRDT to build a collaborative experience
1–10 of 70 posts
Re: You don't need a CRDT to build a collaborative experience
#2One of the main points of this article is to "just use locks", which glosses over a lot of technical complications about locking elements within a document. How long is the lock held? Can it be stolen from a user who has gone offline, or is still online but off to lunch, and we _really_ need to make this change before the presentation in an hour? What if the user comes back online and they have changes, but the lock was stolen - how are those changes reconciled to the document?
I am generally in favor of simpler is better, and if there is a way to build a collaborative experience without using CRDTs, then go for it. However, sometimes the cure can be worse than the disease, and solutions like locking may introduce more technical complexity than originally thought.
Re: You don't need a CRDT to build a collaborative experience
#3One of the main points of this article is to "just use locks", which glosses over a lot of technical complications about locking elements within a document. How long is the lock held? Can it be stolen from a user who has gone offline, or is still online but off to lunch, and we _really_ need to make this change before the presentation in an hour? What if the user comes back online and they have changes, but the lock…
> How long is the lock held?
For however long the user has a browser focus state on the element seems like a reasonable answer, and submit changes as they are made. However, I don't know how you resolve conflicts of two users simultaneously attempting to acquire a lock.
Re: You don't need a CRDT to build a collaborative experience
#4One of the main points of this article is to "just use locks", which glosses over a lot of technical complications about locking elements within a document. How long is the lock held? Can it be stolen from a user who has gone offline, or is still online but off to lunch, and we _really_ need to make this change before the presentation in an hour? What if the user comes back online and they have changes, but the lock…
By exploring that locking and unlocking mechanism, you will find that the logical conclusion in the end, when enough complexity and edge cases get covered/fixed as bugs, that it turns into a crude form of "CRDT" (where it's not actually consistent, but merges within reason for 99% of use cases).
It might as well have been CRDT from the get go.
Re: You don't need a CRDT to build a collaborative experience
#5That's true for collaborative experience. Crdts are a mechanism to handle eventual consistency (that's even the preface of the paper). If you assume that said collaborative experience is always online, you don't need them, and "using locks" as you described is probably enough.
If you want a mechanism to handle that eventual consistency, it's probably better to reuse their principles rather than reinventing something that will eventually ressemble Crdts.
You mentioned "offline first", I think it's probably a good place to pluck that ib https://www.inkandswitch.com/local-first/
Re: You don't need a CRDT to build a collaborative experience
#6> Ever-growing state: for CRDTs to work well they need to keep a record of both what exists, and what has been deleted (so that the deletes aren’t accidentally added back in later). This means that CRDT state will continually expand. There’s a bunch of magic that CRDT library authors are doing with clever compression techniques to make this problem less-bad, but it’s basically in-escapable. The size of your CRDT state is not purely a function of the size of the state the CRDT represents, but also of the number of updates that state has gone through.
A) This is only the case for certain CRDTs, such as sets that support deletion - so, if you want Set semantics with deletion support, you need two sets, one to that tracks all deletions and one that tracks all insertions.
B) You can garbage collection your sets. They don't have to grow forever.
> Complex implementations: CRDTs are easy to implement wrong, so probably don’t roll your own.
Personally, I've never done this. I've just added a `merge(&mut self, other: &Self)` method to structs in Rust. Guaranteeing CRDT properties is often trivial, or at least it was in my case.
> Opaque state: Because the CRDT has to represent both the underlying state and the updates that led to that state
Again, this is only if you need specific operations on your CRDTs and if your CRDTs are encoded in specific ways.
I've said it before, but a trivial crdt looks like this
struct Grows(u64);
impl Grows {
fn merge(&mut self, other: &Self) {
self.0 = max(self.0, other.0);
}
}
> So maybe you are convinced that CRDTs are not the be-all-and-end-all of collaboration, and that you aren’t in one of the two categories where you probably should use a CRDT, and you’ve made it this far in the post.
I am convinced that CRDTs are not the be-all-and-end-all, because Strong Eventual Consistency does not provide strong enough guarantees for all use cases.
Once again we have a CRDT article that's about user collaboration, which I find somewhat frustrating because CRDTs can be used in far more places than that, and user collaboration is like the most complicated thing you could ever write since it's all of the problems of a distributed system and then we add humans into the mix. There is no "good" solution to this problem - CRDTs aren't going to solve it, and neither is any other algorithm, because it's not possible to encode every possible state update in a way that never conflicts and is also what a human expects (especially since humans have varying expectations).
The algorithm/ approach, as described, seems perfectly fine - it will have edge cases just like CRDTs will. In reality, for such an impossibly complex problem, you're probably going to end up with something really complex to solve it. You're almost certainly going to start adding CRDT-like operations, like "ok technically this user held a lock on X, but the other user performed an operation on X that technically commutes, so we can allow both" to alleviate some of the inherent complexities (and UX issues) with locking.
Re: You don't need a CRDT to build a collaborative experience
#7One of the main points of this article is to "just use locks", which glosses over a lot of technical complications about locking elements within a document. How long is the lock held? Can it be stolen from a user who has gone offline, or is still online but off to lunch, and we _really_ need to make this change before the presentation in an hour? What if the user comes back online and they have changes, but the lock…
> How long is the lock held? For however long the user has a browser focus state on the element seems like a reasonable answer, and submit changes as they are made. However, I don't know how you resolve conflicts of two users simultaneously attempting to acquire a lock.
> However, I don't know how you resolve conflicts of two users simultaneously attempting to acquire a lock.
It turns out you just have to pick one... This all depends on a source of truth, and when you are there it's easy to pick one, say based on whichever arrived at the network interface first.
Re: You don't need a CRDT to build a collaborative experience
#8One of the main points of this article is to "just use locks", which glosses over a lot of technical complications about locking elements within a document. How long is the lock held? Can it be stolen from a user who has gone offline, or is still online but off to lunch, and we _really_ need to make this change before the presentation in an hour? What if the user comes back online and they have changes, but the lock…
> How long is the lock held? For however long the user has a browser focus state on the element seems like a reasonable answer, and submit changes as they are made. However, I don't know how you resolve conflicts of two users simultaneously attempting to acquire a lock.
The server must keep track of the locks, and it can only know about the lock being released if the client tells it. E.g by sending a message that the field is not focused any more. The tricky thing is in the "edge" cases, or really the non-perfect cases, which there are plenty of (as I think GP described).
The server can decide that the client is offline if the server misses expected heartbeat messages from the client. But how often will those be sent and how long grace period will we allow? If it's too short then it will be unreliable on shaky 4G connections, if it's too long then it will be annoying in the other direction.
And that's not considering the "social" problems with locks. I've worked on replacing a system that was lock-based with CRDTs where the lunch scenario from MontagFTB actually was a common occurrence.
In an "ideal" scenario your lock acquisition problem is not hard. Client's just show the UI optimistically and whoever the server decide was first gets the lock. The loosing client throws any state the user created in the short time-frame. Over reliable and fast connections for granular locks, this works fine. But in the real world that's just one of the issues with a lock based approach…
Re: You don't need a CRDT to build a collaborative experience
#9You may not need CRDT per-se, but building a collaborative experience is so difficult. I worked on collaborative systems for a bit, and also have read a bit about how Figma and Notion do it (this is a good read: https://www.figma.com/blog/how-figmas-multiplayer-technology...) -- it's still super hard to get right.
This talk by Karri about Linear's "sync engine" is also a good watch: https://www.youtube.com/watch?v=Wo2m3jaJixU.
Re: You don't need a CRDT to build a collaborative experience
#10I agree broadly with the article’s position but I think locks are more harmful than helpful. When I was a Quip user (2018) it was super frustrating to get locked out of a paragraph because someone’s cursor idled there. Instead just allow LWW overwrites. If users have contention and your sync & presence is fast, they’ll figure it out pretty quick, and at most lose 1-2 keystrokes, or one drag gesture, or one color pick.
Notion is “collaborative” and we don’t use a CRDT for text, it’s all last-write-wins decided by the server. However our LWW texts are individually small - one block/paragraph in size - and adding/moving/removing blocks is intention-preserving if not perfectly convergent.
As the article says, the downside for LWW is that “offline” / async collaboration isn’t so great. That’s why we’re working on switching to CRDT for our texts. If you’re interested in bringing CRDTs to a product with a lot of users, consider joining Notion’s Docs team - https://boards.greenhouse.io/notion/jobs/5602426003 / @jitl on Twitter / jake@makenotion.com