Great write-up!
You should check out SQLite-Sync (https://github.com/sqliteai/sqlite-sync) - it might be exactly what you're looking for.
There's also a WASM build (https://www.npmjs.com/package/@sqliteai/sqlite-sync-wasm)
Lessons learned from building a sync-engine and reactivity system with SQLite
31–40 of 41 posts
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#32I must be a bit dense on this little Sunday morning but I don't quite get the problem solved. Is it an issue with reactivity between browser tabs? Hence the use of the Broadcast Channel API? The reactivity being implemented in svelte, what is different from any normal app? could have been react with mobx, zustand, recoil, etc but that's not too relevant. This is just how frontend frameworks are nowadays. Especially s…
There’s a whole world of syncing databases when they’re offline and then reconnect. Local first. If your websocket connection is offline you have to store the incremental changes to your data and send them later on reconnect. This gets complex fast. A lot of these tools (and what the author wrote) offer toolkits to do this well, not having to implement and track all these changes in state manually. In this case the a…
oh ok. What does it require? CRDTs?
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#33Earlier quoted context omitted.
I built my own offline capable, multiplayer capable sync engine with pglite and electric https://github.com/evelant/synchrotron It is opinionated and not for every use case, also very experimental, but you might find some of the ideas interesting.
Oh cool! I'll absolutely take a look.
I'd love to hear what you think of the idea. I really like what electric is building. I'm actually hacking on getting pglite to run on react-native right now =)
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#34Earlier quoted context omitted.
There’s a whole world of syncing databases when they’re offline and then reconnect. Local first. If your websocket connection is offline you have to store the incremental changes to your data and send them later on reconnect. This gets complex fast. A lot of these tools (and what the author wrote) offer toolkits to do this well, not having to implement and track all these changes in state manually. In this case the a…
oh ok. What does it require? CRDTs?
Among other things, but even with a “latest change wins” method you still have to make sure all transactions arrive, data doesn’t stay out of synch forever, and other challenges.
ElectricSql, instantdb, rxdb, jazz.tools, are some of the things I’ve been looking into to make these tasks easier.
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#35Earlier quoted context omitted.
Oh cool! I'll absolutely take a look.
I'd love to hear what you think of the idea. I really like what electric is building. I'm actually hacking on getting pglite to run on react-native right now =)
Do drop into the discord and let me know who you are there. Would love to hear your take on react-native support.
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#36Nice article! An interesting tool that matches the requirements mentioned in the article is Evolu[0] It's a sync engine with e2e encryption based on SQLite. The local-first landscape is quite wide now, and there is probably a solution ready for all kind of needs[1] Building a sync engine can be a nice learning experience, but for production software it's better to pick something that has already faced and resolved al…
Oh wow, thanks for linking evolu! To bad it's typescript only... I'm looking for an e2ee sqlite sync for kotlin. Are you aware of any solutions?
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#37Nice write-up! Would love to read more about practical experience on sync engines. I am also currently building one for a niche use case: a CRDT based smart home system w/o central component: syncing config and state across several embedded/MCU + mobile via CAN bus and BLE, no Internet. It features a simple ORM and persistence and for dev/debug it runs on macos/Linux. Written in C++ and Dart. Due to memory and other…
I personally don't have use for the insight to this "obscure" tech stack but typically the more obscure the tech stack the more people can benefit from you sharing your perspective.
If someone shares their additional take on rewriting in rust (I say this with love as the type of person to do that) it doesn't add as much value because many of those experiences are shared regularly. If you share your unique tech stack, the people in the woodwork who might be struggling to do something along those lines but can't find the resources will benefit way more.
So you should!
Also can you elaborate on the CAN bus part of your smart home? Is your car wired into your home network?
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#38performance degradation observed using the first approach at high concurrency recently discussed here: https://news.ycombinator.com/item?id=44490510
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#39Do you cache data on the client and write buffer it on the way back to the server? I made a key value store as a hobby project, that also kind of streamed the data and it was convenient to do that .
Re: Lessons learned from building a sync-engine and reactivity system with SQLite
#40This is great to see and I like the simplicity of the approach.
You can also look at PowerSync (which I work on). It's in a similar space as ElectricSQL. It syncs to SQLite on the client-side and provides built-in reactivity. On the web, it uses wa-sqlite with either OPFS or IndexedDB. It also takes care of things like multi-tab on web, and queueing and uploading client-side mutations to the backend.