Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

41–50 of 131 posts

Re: Stop syncing everything

#41

Post by gbalduzzi »

matlin wrote: post #8

My ideal version of this is simple: just define the queries you want (no matter how complex) and the you'll get exactly the data you need to fulfill those queries, no more, no less. And the cherry on top would be to have your queries update automatically with changes both locally and remote in close to real-time. That's basically what we're doing with Triplit ( https://triplit.dev ), be it, not with SQL--which is a p…

> just define the queries you want (no matter how complex) and the you'll get exactly the data you need to fulfill those queries, no more, no less

So GraphQL?

Re: Stop syncing everything

#42

Post by theSherwood »

This is a very interesting approach. Using pages as the basic sync unit seems to simplify a lot. It also makes the sync of arbitrary bytes possible. But it does seem that if your sync is this coarse-grained that there would be lots of conflicting writes in applications with a lot of concurrent users (even if they are updating semantically unrelated data). Seems like OT or CRDT would be better in such a use-case. I'd be interested to see some real-world benchmarks to see how contention scales with the number of users.

Re: Stop syncing everything

#43

Post by ncruces »

chacham15 wrote: post #38

So, if I understand correctly, the consistency model is essentially git. I.e. you have a local copy, makes changes to it, and then when its time to "push" you can get a conflict where you can "rebase" or "merge". The problem here is that there is no way to cleanly detect a conflict. The documentation talks about pages which have changed, but a page changing isnt a good indicator of conflict. A conflict can happen due…

They address this later on.

If strict serializability is not possible, because your changes are based on a snapshot that is already invalid, you can either replay (your local transactions are not durable, but system-wide you regain serializability) or merge (degrading to snapshot isolation).

As long as local unsynchronized transactions retain the page read set, and look for conflicts there, this should be sound.

Re: Stop syncing everything

#44

Post by stavros »

I didn't go into the implementation details so I won't comment on that, but I will say that this is a really important problem to solve. I've long wanted/needed an agnostic sync layer that apps could use to sync my changes across clients and servers, and it would be great if app developers didn't have to put any work into it, they'd just point their database to a URL and the database would take care of all the syncing.

This means that the app developer could support private syncing without any effort, and I wouldn't have to pay a subscription just for sync, I could just run a sync server on my NAS and all apps would know to sync their data with it. Of course, app developers don't want this because subscriptions are a reliable income stream, but hopefully users would push for it.

Re: Stop syncing everything

#45

Post by Cthulhu_ »

stavros wrote: post #44

I didn't go into the implementation details so I won't comment on that, but I will say that this is a really important problem to solve. I've long wanted/needed an agnostic sync layer that apps could use to sync my changes across clients and servers, and it would be great if app developers didn't have to put any work into it, they'd just point their database to a URL and the database would take care of all the syncin…

~10 years ago, CouchDB was the answer to this problem. I don't know what the status of that is in 2025, but at the time it was used to e.g. sync airplane sales tablets with a central database once they were back on the ground and online.

Re: Stop syncing everything

#46

Post by stavros »

Cthulhu_ wrote: post #45
stavros wrote: post #44

I didn't go into the implementation details so I won't comment on that, but I will say that this is a really important problem to solve. I've long wanted/needed an agnostic sync layer that apps could use to sync my changes across clients and servers, and it would be great if app developers didn't have to put any work into it, they'd just point their database to a URL and the database would take care of all the syncin…

~10 years ago, CouchDB was the answer to this problem. I don't know what the status of that is in 2025, but at the time it was used to e.g. sync airplane sales tablets with a central database once they were back on the ground and online.

Yeah, what happened to CouchDB? I thought it would be much more popular, but I haven't heard of it in years. Maybe this problem isn't actually important, or maybe people don't know they can solve it in this way?

Re: Stop syncing everything

#47

Post by wim »

Looks impressive! Using the VFS is such a fun "hack" :)

We developed our own sync engine for an offline-first IDE for notes/tasks [1] we're building, where the data structure is a tree (or graph actually) to support outlining operations. Conflict resolution is always the challenge, and especially with trees multiple offline players can optimistically commit local changes which would result in an invalid tree state once globally merged.

The way we handle this is by rolling back tree inserts when a client comes online and receives other tree insert deltas. From what I understand from the description of SQLSync, the best way to handle this would be to pull in a latest snapshot and then replay. Pulling in a full snapshot sounds rather expensive though. We have some sort of heuristic where we can do this if the number of incoming deltas would be very large, but for most offline syncing we never need it. Just curious how SQLSync defines these snapshots? Sticking with the note-taking example, in our case we can't really have a snapshot of a single "note" because of graph features like transclusions. Does SQLSync have some clever way to avoid having to send all state in order to "reset and replay"?

[1] https://thymer.com

Re: Stop syncing everything

#48

Post by anentropic »

ccorcos wrote: post #33

How are permissions supposed to work? Suppose a page has data that I need to see and also has data I can’t see. Does this mean I need to demoralize my entire data model?

You probably want a db-per-user architecture with this kind of thing

Re: Stop syncing everything

#49

Post by krick »

Maybe it's just me, since people here in the comments apparently understand what this is, but even after skimming the comments, I don't. Some simplified API example would be useful either in the "marketing post" or (actually, and) in the github readme.

I mean, it's obviously about syncing stuff (despite the title), ok. It "simplifies the development", "shares data smoothly" and all the other nice things that everything else does (or claims to do). And I can use it to implement everything where replication of data might be useful (so, everything). Cool, but... sorry, what does it, exactly?

The biggest problem with syncing is, obviously, conflict resolution. Graft "doesn’t care about what’s inside those pages", so, obviously, it cannot solve conflicts. So if I'm using it in a note-taking app, as suggested, every unsynced change to a plain text file will result in a conflict. So, I suppose, it isn't what it's for at all, it's just a mechanism to handle replication between 2 SQLite files, when there are no conflicts between statements (so, what MySQL or Postgres do out of the box). Right? So, it will replace the standard SQLite driver in my app code to route all requests via some Graft-DB that will send my statements to external Graft instance as well as to my SQLite storage? Or what?

Re: Stop syncing everything

#50

Post by bbrks »

stavros wrote: post #46
Cthulhu_ wrote: post #45

Earlier quoted context omitted.

~10 years ago, CouchDB was the answer to this problem. I don't know what the status of that is in 2025, but at the time it was used to e.g. sync airplane sales tablets with a central database once they were back on the ground and online.

Yeah, what happened to CouchDB? I thought it would be much more popular, but I haven't heard of it in years. Maybe this problem isn't actually important, or maybe people don't know they can solve it in this way?

This space is very much alive and well and I'm really glad there's more competition cropping up from smaller projects like OP.

Disclaimer: I have worked on a sync backend for a company in this space for the last 8 years. You can probably find out where if you look, but this comment won't be a sales pitch.

Competition like this has incredible value for communities with poor internet access but reasonable development capabilities. Think about travelling doctors in the lesser developed countries and areas of Africa for instance. Quite often entire villages get medical checkups done, with data being recorded on a tablet. This can be synced once the doctor gets to a town with internet access for follow-up. Of course, projects like the above do not have big budgets. Unfortunately they are priced out of using a lot of tech to solve these problems (my company included in this statement)

On the more enterprise-y side, which is where I mostly sit, a lot of airlines, cruise ships, retail and field-based industry use these technologies, since they are prone to periods of being completely offline or in a P2P mesh only. Cloud databases or even regular replicated databases running in-situ are a non-starter, since there won't be anybody around to administer it. Replication is a difficult problem at the best of times, let alone in periods of comms downtime.

Post reply on HN