Interesting - a lot of problems I face seem to fit into this general framework, and I feel like I keep reinventing the wheel. How is conlict resolution handled?
LiveStore: State management based on reactive SQLite and built-in sync engine
21–30 of 46 posts
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#22Sounds very cool. Not impressed with the cross platform claim when the first thing I see is that android web is not supported, though.
That's a good point. I'm in touch with the Android/Chrome team about the underlying issue. I was hoping the underlying Android web issue would have been fixed by now (as I first noticed it ~3 years ago with some indication for progress), but looks like LiveStore needs a custom workaround for it. You can track the progress here: https://github.com/livestorejs/livestore/issues/321 I hope you understand that bridging th…
I’ve been working on a very similar sync problem and hit this too. I think the way forward is to use a broadcast channel to elect an individual worker that communicates with all other contexts using the broadcast channel.
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#23Earlier quoted context omitted.
I have evaluated the local first landscape and few solutions are as neat as this one! The only other (more mature seems to be) is tinyBase but that's based on a different model (CRDTs vs Event sourcing) Few questions: Would it not be possible to store and rely on larger amount of data on SQLite and keeping it on disk as an option? I see that there's a limitation of 1 GB. So maybe just a config flip to change the mode…
curious: did you literally use the Local-first Landscape[1] to do the evaluation? (full disclosure: co-author of landscape) [1] https://localfirst.fm/landscape
Yes already did through most of them. Either we have libraries around CRDTs or we have full BaaS (think Parse if anyone remembers or Firebase) soft of things that possibly can be self deployed but have a very heavy and iron clad curtain on the underlying storage medium (Postgres/SQLite) etc exposing a world view of their own.
tinyBase seems to be the only one that stands out in that regard or this LiveStore now.
PS: Thank you for putting this landscape. That really has enabled me or anyone else to look at what options are there. No amount of googling would do that kind of compilation. I find it very useful.
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#24This looks like interesting technology! Congrats on the launch, it's great to have people exploring the space of realtime data storage & sync. Some thoughts, that I hope you can find constructive.
The landing page draws comparison to Figma, Linear and Notion. But they are vastly different use cases.
Figma is document-centric, which means that:
- All data is tied to a single document, limiting their size.
- Requires that single backend session holds a document in-memory and deals with split-brain issues.
- Operational complexity arises around deployment.
- User interactions are often continuous (one event / frame), imposing tighter latency requirements.
- Generally requires that the document stays loaded in a memory in a stateful backend session.
- Technologies like Jamsocket are targeting such use cases.
Notion & Linear, on the other hand, have collaborative editing but not to the same degree of realtimeness. - Data is not tied to a single document.
- Data can grow unbounded.
- Data is typically more relational.
Presenting both these uses cases adds confusion. By using SQLite (emphasis: database and "SQL"), it makes the technology appear at first glance more suited for the Linear/Notion use case. However, the opposite appears to be true after reading https://docs.livestore.dev/evaluation/when-livestore/.
- "All the client app data should fit into a in-memory SQLite database".
- "Reasons when not to use LiveStore" -> "Your app data is highly connected across users".
The Figma-like document use case does seem like something that LiveStore could support. When it comes to designing a data model for collaborative apps (CRDT or CRDT-like), the most simple and flexible solution is to store every object as a map of [Object ID] -> [Property Name] -> [Property Value]. Assuming that the properties names & values come from a fixed set and are typed (this will generally be true unless you allow for arbitrary user-defined fields), that looks like a database row! So why not just store it in a database, indeed.
However, among databases, there are object stores and relational databases. The latter is useful, as applications often want to represent relationships between different objects. But if we support multiplayer-like use cases which implies the absence of server-side transactions, how should conflicting events involving the creation/deletion/relationships between objects be handled? Is it entirely on the application to think about it? This is an interesting topic that I believe LiveStore seems well-positioned to innovate in.
I also find the "local-first" emphasis to be at odds with all of Figma/Notion/Linear. Local-first software tends to have a limited number of concurrent users accessing the data in-mind. SaaS applications that are truly targeted at collaboration tend to have a lot of different requirements. On the other hand, those applications do tend to have relatively limited offline support, which a local-first application tends to emphasize more.
The Figma-like use cases also has additional requirements. But I understand this is beta and look forward to seeing further development!
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#25...incurabel...next!
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#26seems you are using wa-sqlite. Can you elaborate on your strategy for persisting data offline? are you using opfs(which variant AccessHandlePoolVFS?) or indexddb? Also how are you mitigating against unstable implementation of opfs in different browsers and 7 day retention policy of safari for indexeddb?
since sqlite now also provides official wasm builds why did you decided to go with wa-sqlite?
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#27Landing page is awesome! Love the way you demo the software. This is a crowded space with Zero/Convex/Tinybase etc and others so this is a great way to stand out.
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#28This looks like interesting technology! Congrats on the launch, it's great to have people exploring the space of realtime data storage & sync. Some thoughts, that I hope you can find constructive. The landing page draws comparison to Figma, Linear and Notion. But they are vastly different use cases. Figma is document-centric, which means that: - All data is tied to a single document, limiting their size. - Requires t…
I worked with Rudi at Figma and of course support his comment - Figma seems to be mentioned for marketing, not for the actual technical comparison.
For others looking for more details on how Figma's sync engines differ and why 2 sync engines emerged, I had a long thread about it here:
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#29Hi folks, creator of LiveStore here (prev. founder Prisma). Very excited to launch LiveStore in beta today after having worked on it over the past 4 years. I've built it for myself working on Overtone, an ambitious music client aiming for a native-grade high-performance app feel. LiveStore embraces SQLite by adding a signals-based reactivity layer and combines it with event-sourced based syncing (similar to Git). Hap…
> LiveStore extends its local event-sourcing model globally by synchronizing events across all clients, typically through a central sync backend.
Is that a requirement or could there also be federated nodes, or a full p2p modus operandus? Think for use in decentralized social networking.
Re: LiveStore: State management based on reactive SQLite and built-in sync engine
#30Earlier quoted context omitted.
That's a good point. I'm in touch with the Android/Chrome team about the underlying issue. I was hoping the underlying Android web issue would have been fixed by now (as I first noticed it ~3 years ago with some indication for progress), but looks like LiveStore needs a custom workaround for it. You can track the progress here: https://github.com/livestorejs/livestore/issues/321 I hope you understand that bridging th…
I’ve been working on a very similar sync problem and hit this too. I think the way forward is to use a broadcast channel to elect an individual worker that communicates with all other contexts using the broadcast channel.
fyi, I've done something similar and the cleanest method for leader election is to use the web locks api:
https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_A...
It is supported by all browsers. You just have all the tabs try to lock the same resource and have them return a promise in the lock callback. The first one wins and when that tab closes the next one in line gets automatically elected.
The leader can then use a broadcast channel and act as a server for all the other tabs to serialize access to any shared resources.