Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

71–80 of 131 posts

Re: Stop syncing everything

#71

Post by imagio »

carlsverre wrote: post #64
imagio wrote: post #61

Very interesting! I've been hacking on a somewhat related idea. I'm prototyping a sync system based on pglite and the concept of replicating "intentions" rather than data. By that I mean replicating actions -- a tag and a set of arguments to a business logic function along with a hybrid logical clock and a set of forward & reverse patches describing data modified by the action. As long as actions are immutable and an…

Glad you enjoyed Graft! The system your describing sounds very cool! It's actually quite similar to SQLSync. The best description of how SQLSync represents and replays intentions (SQLSync calls them mutations) is this talk I did at WasmCon 2023: https://www.youtube.com/watch?v=oLYda9jmNpk

Cool! That's an interesting approach putting the actions in wasm. I'm going for something more tightly integrated into an application rather than entirely in the database layer.

The actions in my prototype are just TS functions (actually Effects https://effect.website/ but same idea) that can arbitrarily read and write to the client local database. This does put some restrictions on the app -- it has to define all mutations inside of actions and capture any non-deterministic things other than database access (random number, time, network calls, etc) as part of the arguments. Beyond that what an app does inside of the actions can be entirely arbitrary.

I think that hits the sweet spot between flexibility, simplicity, and consistency. The action functions can always handle divergence in whatever way makes sense for the application. Clients will always converge to the same semantically valid state because state is always advanced by business logic, not patches.

Patches are recorded but only for application to the server's database state and for checking divergence from expected results when replaying incoming actions on a client. It should create very little load on the backend server because it does not need to execute action functions, it can just apply patches with the confidence that the clients have resolved any conflicts in a way that makes the most sense.

It's fun and interesting stuff to work on! I'll have to take a closer look at SQLSync for some inspiration.

Re: Stop syncing everything

#72

Post by whizzter »

ozim wrote: post #59
vladstudio wrote: post #57

in case anyone finds this useful, here's the slowly growing collection of links to similar tools: https://tinybase.org/ https://www.evolu.dev/ https://replicache.dev/ https://fireproof.storage/ https://vlcn.io/ https://www.instantdb.com/ https://loro.dev/ https://electric-sql.com/ https://docs.y-sweet.dev/ https://syncedstore.org/docs/ https://collabs.readthedocs.io/en/latest/ https://remotestorage.io/ https://rxdb.i…

I wonder why there are so many, just people reinventing stuff that no one really needs? For me personally I have 4 of those as visited, pouchdb, automerge, loro and sqlsync of course. I was trying to fit such a tool into existing architectures that I deal with at work but nothing really makes sense. My guess is those solutions are in totally wrong abstraction layer, creators think that would be best thing since slice…

It's about the multiplayer application case, think Google Write/Sheets/etc. Applications with data that can change by multiple users and you can both see it live and the application (that keeps state in memory/localdb) is also resilient to disconnects.

The reason people descend into this madness is because visible replication code is tricky and the general feeling is that it'll infect parts that shouldn't be infected (or at least not without a general framework).

So at a somewhat trivial level you have:

A: A bare log replication system (where the application needs awareness for most object types multiplying object complexity).

B: A object replication system where the framework handles all object types coherently and the application "only" needs to be aware of how to fetch objects, think a KV store that stores fairly basic objects with some addressing support.

C: Since recent crowd "wisdom" dictates that most KV stores will likely re-implement SQL functionality badly, people go straight to doing the SQL case (maybe they've had a curiosity about SQL databases already that they're scratching)

I've recently built A (basic replication and LWW) and building the application I'm definitively feeling an itch to just start over or adjust to support B (a simple replicated KV store) to separate the concerns more, I can see how I would also feel the SQL itch of C (but having done SQL like systems before it's not as bad for me).

For this first application A will suffice (since the offline needs are relatively trivial) but having a more complicated application in mind I'm strongly considering B for that project (along with designs or third party libs to make it happen).

I think a big gap in the space is that most seem focused on "documents" (esp the CRDT based tools), ie a document being the atomic piece that is synchronized but imo it leaves a big gap in that now all regular application management tools like SQL query tools are useless since essentially you only have a bunch of "blobs" or worse. If you want the regular enterprise SQL backend these tools don't seem to have a focus on synchronizing to those regular backend storage systems.

Re: Stop syncing everything

#73

Post by eatonphil »

> [rqlite and dqlite] are focused on increasing SQLite’s durability and availability through consensus and traditional replication. They are designed to scale across a set of stateful nodes that maintain connectivity to one another.

Little nitpick there, consensus anti-scales. You add more nodes and it gets slower. The rest of the section on rqlite and dqlite makes sense though, just not about "scale".

Re: Stop syncing everything

#74

Post by ncruces »

bastawhiz wrote: post #69
ncruces wrote: post #43

Earlier quoted context omitted.

They address this later on. If strict serializability is not possible, because your changes are based on a snapshot that is already invalid, you can either replay (your local transactions are not durable, but system-wide you regain serializability) or merge (degrading to snapshot isolation). As long as local unsynchronized transactions retain the page read set, and look for conflicts there, this should be sound.

> your local transactions are not durable This manifests itself to the user as just data loss, though. You do something, it looks like it worked, but then it goes away later.

From the description, you can reapply transactions. How the system handles it (how much of it is up to the application, how much is handled in graft) I have no idea.

Re: Stop syncing everything

#75

Post by giancarlostoro »

Reading their readme:

> Licensed under either of

> Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)

> MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

> at your option.

I see this now and then, but it makes me wonder, why would I pick in this case Apache over MIT? Or is this software actually Apache licensed, but the developer is giving you greenlight to use it under the terms of the MIT? But at that point I don't get why not just license it all under MIT to begin with...

Re: Stop syncing everything

#76

Post by CorrectHorseBat »

giancarlostoro wrote: post #75

Reading their readme: > Licensed under either of > Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0 ) > MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT ) > at your option. I see this now and then, but it makes me wonder, why would I pick in this case Apache over MIT? Or is this software actually Apache licensed, but the developer is giving you greenlight to us…

One thing I can think of is that Apache gives you a patent grant and MIT doesn't

Re: Stop syncing everything

#77

Post by carlsverre »

giancarlostoro wrote: post #75

Reading their readme: > Licensed under either of > Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0 ) > MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT ) > at your option. I see this now and then, but it makes me wonder, why would I pick in this case Apache over MIT? Or is this software actually Apache licensed, but the developer is giving you greenlight to us…

Per the Rust FAQ [1]:

> The Apache license includes important protection against patent aggression, but it is not compatible with the GPL, version 2. To avoid problems using Rust with GPL2, it is alternately MIT licensed.

The Rust API guidelines also recommend the same: https://rust-lang.github.io/api-guidelines/necessities.html#...

[1]: https://github.com/dtolnay/rust-faq#why-a-dual-mitasl2-licen...

From my perspective (as the author of Graft) my goal was to be open source and as compatible as possible with the Rust ecosystem. Hence the choice to dual license.

Re: Stop syncing everything

#78

Post by giancarlostoro »

carlsverre wrote: post #77
giancarlostoro wrote: post #75

Reading their readme: > Licensed under either of > Apache License, Version 2.0 (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0 ) > MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT ) > at your option. I see this now and then, but it makes me wonder, why would I pick in this case Apache over MIT? Or is this software actually Apache licensed, but the developer is giving you greenlight to us…

Per the Rust FAQ [1]: > The Apache license includes important protection against patent aggression, but it is not compatible with the GPL, version 2. To avoid problems using Rust with GPL2, it is alternately MIT licensed. The Rust API guidelines also recommend the same: https://rust-lang.github.io/api-guidelines/necessities.html#... [1]: https://github.com/dtolnay/rust-faq#why-a-dual-mitasl2-licen... From my perspect…

Thank you! That is helpful, and understandable from that context. I to try to follow the best standards surrounding the language I use for a given project.

Re: Stop syncing everything

#79

Post by carlsverre »

eatonphil wrote: post #73

> [rqlite and dqlite] are focused on increasing SQLite’s durability and availability through consensus and traditional replication. They are designed to scale across a set of stateful nodes that maintain connectivity to one another. Little nitpick there, consensus anti-scales. You add more nodes and it gets slower. The rest of the section on rqlite and dqlite makes sense though, just not about "scale".

Hey Phil! Also you're 100% right. I should use a different word than scale. I was meaning scale in the sense that they "scale" durability and availability. But obviously it sounds like I say they are scaling performance.

I've changed the wording to "They are designed to keep a set of stateful nodes that maintain connectivity to one another in sync.". Thank you!

Re: Stop syncing everything

#80

Post by eatonphil »

carlsverre wrote: post #79
eatonphil wrote: post #73

> [rqlite and dqlite] are focused on increasing SQLite’s durability and availability through consensus and traditional replication. They are designed to scale across a set of stateful nodes that maintain connectivity to one another. Little nitpick there, consensus anti-scales. You add more nodes and it gets slower. The rest of the section on rqlite and dqlite makes sense though, just not about "scale".

Hey Phil! Also you're 100% right. I should use a different word than scale. I was meaning scale in the sense that they "scale" durability and availability. But obviously it sounds like I say they are scaling performance. I've changed the wording to "They are designed to keep a set of stateful nodes that maintain connectivity to one another in sync.". Thank you!

Thank you!
Post reply on HN