Live data from Hacker News

Stop syncing everything

sqlsync.dev

61–70 of 131 posts

Re: Stop syncing everything

#61
Very interesting! I've been hacking on a somewhat related idea. I'm prototyping a sync system based on pglite and the concept of replicating "intentions" rather than data. By that I mean replicating actions -- a tag and a set of arguments to a business logic function along with a hybrid logical clock and a set of forward & reverse patches describing data modified by the action.

As long as actions are immutable and any non-deterministic inputs are captured in the arguments they can be (re)executed in total clock order from a known common state in the client database to arrive at a consistent state regardless of when clients sync. The benefit of this I realized is that it works perfectly with authentication/authorization using postgres row level security. It's also efficient, letting clients sync the minimal amount of information and handle conflicts while still having full server authority over what clients can write.

There's a lot more detail involved in actually making it work. Triggers to capture row level patches and reverse patches in a transaction while executing an action. Client local rollback mechanism to resolve conflicts by rolling back local db state and replaying actions in total causal order. State patch actions that reconcile the differences between expected and actual outcomes of replaying actions (for example due to private data and conditionals). And so on.

The big benefits of this technique is that it isn't just merging data, it's actually executing business logic to move state forward. That means it captures user intentions where a system based purely on merging data cannot. Traditional crdt that merges data will end up at a consistent state but can provide zero guarantees about the semantic validity of that state to the end user. By replaying business logic functions I'm seeking to guarantee that the state is not only consistent but maximally preserves the intentions of the user when reconciling interleaved writes.

This is still a WIP and I don't have anything useful to share yet but I think the core of the idea is sound. Exciting to see so much innovation in the space of data sync! It's a tough problem and no solution (yet) handles the use cases of many different types of apps.

Re: Stop syncing everything

#62
post #52
post #46

Earlier quoted context omitted.

Yeah, what happened to CouchDB? I thought it would be much more popular, but I haven't heard of it in years. Maybe this problem isn't actually important, or maybe people don't know they can solve it in this way?

Not an expert on CouchDB, but used it recently in a project and loved the syncing with the corresponding pouchdb at the frontend. My main problem, what was missing imho was security: Interactions with couchdb where directly with the database, ie no intermediate API available. Maybe that was a good idea when couchDB was designed, but it made me feel uncomfortable in 2024. If anyone knows of an intermediate API-Layer f…

Interesting that you're seeing the lack of intermediate server/API as a drawback. InstantDB sees it as progress, see their essay on it:

https://www.instantdb.com/essays/sync_future

Re: Stop syncing everything

#63

Hey friends! Author of Graft here. Just want to say, huge thanks for all the great comments, stars, and support. Feels really nice to finally be building in public again. I'm going to force myself to sign off for the evening. Will be around first thing to answer any other questions that come up! I just arrived to Washington, DC to attend Antithesis BugBash[1] and if I don't get ahead of the jet lag I'm going to regre…

Thanks for sharing this, it looks really cool. I also wanted to explicitly mention that the graphics are great. IDK if you made them, or got help, but they do a great job explaining your point. It can be hard to create graphics for technical concepts like pages and databases, but these work well.

Thank you! I made them myself using http://excalidraw.com/.

Re: Stop syncing everything

#64
post #61

Very interesting! I've been hacking on a somewhat related idea. I'm prototyping a sync system based on pglite and the concept of replicating "intentions" rather than data. By that I mean replicating actions -- a tag and a set of arguments to a business logic function along with a hybrid logical clock and a set of forward & reverse patches describing data modified by the action. As long as actions are immutable and an…

Glad you enjoyed Graft! The system your describing sounds very cool! It's actually quite similar to SQLSync. The best description of how SQLSync represents and replays intentions (SQLSync calls them mutations) is this talk I did at WasmCon 2023: https://www.youtube.com/watch?v=oLYda9jmNpk

Re: Stop syncing everything

#65
post #47

Looks impressive! Using the VFS is such a fun "hack" :) We developed our own sync engine for an offline-first IDE for notes/tasks [1] we're building, where the data structure is a tree (or graph actually) to support outlining operations. Conflict resolution is always the challenge, and especially with trees multiple offline players can optimistically commit local changes which would result in an invalid tree state on…

Thank you! And I agree haha. I love writing SQLite VFS's.

In SQLSync, the major issue was precisely what you describe: pulling in the full snapshot to replay. This is the main driver behind the "partial" aspect of the Graft design. It means that clients only need to pull the portion of the incoming snapshot that they don't already have and that overlap with the read/write set of their transactions. So yes, to answer your question once SQLSync is powered by Graft, it will frequently be able to avoid downloading all state in order to reset and replay.

Note that if a client is ok with relaxing to snapshot isolation (in particular this means clients may experience Write Skew[1]), and the r/w set of their local transaction does not intersect the snapshot changeset, Graft is able to perform an automatic merge.

[1]: https://jepsen.io/consistency/phenomena/a5b

Re: Stop syncing everything

#67

This approach has a problem when the mobile client is on the end of a very slow connection, yet the diff that needs to be synced is gigabytes, perhaps because it's an initial sync or the database got bloated for whatever reason. A hybrid approach is to detect slow syncing (for example when sync hasn't completed after 5 seconds), and instead send queries directly to the server because there is a good chance the task t…

This is a great point. Solutions like Graft which involve syncing data to the edge work poorly when the dataset size is too large and not partitioned enough to be consistency partial.

This is why Graft isn't just focused on client sync. By expanding the focus to serverless functions and the edge, Graft is able to run the exact same workload on the exact same snapshot (which it can validate trivially due to its consistency guarantees) anywhere. This means that a client's workload can be trivially moved to the edge where there may be more resources, a better network connection, or existing cached state.

Re: Stop syncing everything

#68
post #56

Man this looks super awesome. I will be extremely excited to ditch CouchDB for this (even tho I'm an Erlang fan). I'll certainly be keeping an eye on the project and I'll pitch in where I can!

Thank you!!

From an extremely brief scan, it appears that Erlang wrappers around SQLite should be able to use the Graft SQLite extension just fine.

Alternatively, it would be reasonably straight forward to wrap Graft Client (Rust library) directly in an Erlang NIF using something like https://github.com/rusterlium/rustler

Let's make it happen! :)

Re: Stop syncing everything

#69
post #43

So, if I understand correctly, the consistency model is essentially git. I.e. you have a local copy, makes changes to it, and then when its time to "push" you can get a conflict where you can "rebase" or "merge". The problem here is that there is no way to cleanly detect a conflict. The documentation talks about pages which have changed, but a page changing isnt a good indicator of conflict. A conflict can happen due…

They address this later on. If strict serializability is not possible, because your changes are based on a snapshot that is already invalid, you can either replay (your local transactions are not durable, but system-wide you regain serializability) or merge (degrading to snapshot isolation). As long as local unsynchronized transactions retain the page read set, and look for conflicts there, this should be sound.

> your local transactions are not durable

This manifests itself to the user as just data loss, though. You do something, it looks like it worked, but then it goes away later.

Post reply on HN