Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lessons learned from building a sync-engine and reactivity system with SQLite

finkelstein.fr

Post reply on HN
Print view View original

11–20 of 41 posts

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#11

Post by antoinefink »

Void_ wrote: post #9
judge123 wrote: post #6

It feels like everyone is solving the sync problem from a slightly different angle. Has anyone here used a few of these in production?

We use very similar solution in https://reflect.app Using the wa-sqlite library, and our own Mobx-based model layer similar to what Linear does. I gave a short talk about it a few months ago https://www.youtube.com/watch?v=b0GKWzeyZko

Thanks for sharing this talk. I didn’t know about MobX, but I did use your fork of wa-sqlite (https://github.com/team-reflect/wa-sqlite/) since I also needed the sqlite-vec extension :) By the way, Reflect is an awesome app; I'm actually a paid subscriber.

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#12

Post by JSR_FDED »

st-keller wrote: post #8

Ok is this about a reactive app with with a local database automatically synced to a remote db? All fully encrypted (at rest and in transit)? I thought this is what everyone does nowadays! We built an app like this in 2019 - yes - it was a bit of a challenge with the encryption but the „syncing data“-part is what every litte multiplayer-game has to deal with like forever now. Seems i‘m out of touch with the current s…

The value is in the author’s experience with other tools that caused problems as his database grew, and in learning from his reasoning about the appropriate solution for his particular problem.

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#14

Post by ethan_smith »

boris wrote: post #7

> And because Electric syncs every change granularly, you are certain that the state of your local database is exactly the same as the server's. I don't see how this certainty follows from "granularity" (whatever that means in this context). I believe to have such a certainty one would need the synchronization to happen within a single transaction that spans both client and server databases.

Correct - granular syncing alone doesn't guarantee consistency; you'd need either a distributed transaction protocol or a conflict resolution strategy with eventual consistency semantics.

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#15

Post by mockingloris »

Amazing write-up! I’m thrilled to see your exploration of a reactive SQLite sync engine; it resonates deeply with my own journey toward Local‑First, Offline‑Resilient tools aimed at relieving pressure, not just impressing. Keeping all user data local in SQLite while enabling seamless, encrypted syncing and reactivity. A nice segue in pragmatic autonomy.

It mirrors the moment I embraced tools like Obsidian; designing for real-world constraints; valuing simplicity, privacy, and functionality; especially in low‑connectivity environments.

PS: I was hooked on ...SQLite

Bravo!

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#16

Post by mrkeen »

> Selecting a sync engine is a lot about knowing the constraints of your application.

Not so sure about this. These seem more like fundamentals than sliding scales.

> How many people will concurrently edit the same resources?

More than 1.

> How write-heavy is it?

Write-heavy enough that you'll encounter an unexpected write between two reads.

> Can you expect unreliable connections or offline usage?

Yes.

CAP with Electron and SQLite is no different from CAP with Tauri and MySQL.

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#17

Post by boris »

ethan_smith wrote: post #14
boris wrote: post #7

> And because Electric syncs every change granularly, you are certain that the state of your local database is exactly the same as the server's. I don't see how this certainty follows from "granularity" (whatever that means in this context). I believe to have such a certainty one would need the synchronization to happen within a single transaction that spans both client and server databases.

Correct - granular syncing alone doesn't guarantee consistency; you'd need either a distributed transaction protocol or a conflict resolution strategy with eventual consistency semantics.

I would say there is no certainty with eventual consistency, only hope.

Re: Lessons learned from building a sync-engine and reactivity system with SQLite

#18

Post by SkiFire13 »

gczh wrote: post #2

very cool! have you tried Tanstack DB and ElectricSQL? https://x.com/marcelohbairros/status/1956892684859158892

Not sure about Tanstack DB, but from the second paragraph in the article:

> First Try: PGlite and Electric

Post reply on HN