Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

111–120 of 131 posts

Re: Stop syncing everything

#111

Post by fauigerzigerk »

NickM wrote: post #110
fauigerzigerk wrote: post #105

Earlier quoted context omitted.

The problem I have with CRDTs is that while being conflict-free in a technical sense they don't allow me to express application level constraints. E.g, how do you make sure that a hotel room cannot be booked by more than one person at a time or at least flag this situation as a constraint violation that needs manual intervention? It's really hard to get anywhere close to the universal usefulness and simplicity of cen…

Yeah, this is a limitation, but generally if you have hard constraints like that to maintain, then yeah you probably should be using some sort of centralized transactional system to avoid e.g. booking the same hotel room to multiple people in the first place. Even with perfect conflict resolution, you don't want to tell someone their booking is confirmed and then later have to say "oh, sorry, never mind, somebody els…

I agree, hotel booking is not a great example.

I think CRDTs would be applicable to a wider range of applications if it was possible to specify soft constraints.

So after merging your changes you can query the CRDT for a list of constraint violations that need to be resolved.

Re: Stop syncing everything

#112

Post by sanbor »

In the post or the comments here nobody mentions end-to-end encryption. Obsidian Sync offers multi-user sync withe2e encryption. An open source general tools solving the same problem would be great.

Re: Stop syncing everything

#113

Post by isaachinman »

vladstudio wrote: post #57

in case anyone finds this useful, here's the slowly growing collection of links to similar tools: https://tinybase.org/ https://www.evolu.dev/ https://replicache.dev/ https://fireproof.storage/ https://vlcn.io/ https://www.instantdb.com/ https://loro.dev/ https://electric-sql.com/ https://docs.y-sweet.dev/ https://syncedstore.org/docs/ https://collabs.readthedocs.io/en/latest/ https://remotestorage.io/ https://rxdb.i…

Wrote a very-related blog post here:

https://marcoapp.io/blog/offline-first-landscape

Re: Stop syncing everything

#114

Post by drewcoo »

mdavidn wrote: post #85
kiitos wrote: post #34

Earlier quoted context omitted.

I think I'm probably operating with definitions of client and commit that are different than yours. Specifically, I don't really see how a client can "commit locally" and "commit globally" as separate things. I understand a client to be something that interacts with your metastore API, which provides a single "commit" operation, that AFAICT will return success/failure based on local commit state, not global state. Is…

Graft's definition sounds more like a Git "commit" than one found in a SQL standard. Perhaps that's the source of this confusion?

Git is a DVCS. The D stands for distributed, meaning (in old people language) masterless. Git doesn't have a "global."

Re: Stop syncing everything

#115

Post by drewcoo »

ozim wrote: post #59
vladstudio wrote: post #57

in case anyone finds this useful, here's the slowly growing collection of links to similar tools: https://tinybase.org/ https://www.evolu.dev/ https://replicache.dev/ https://fireproof.storage/ https://vlcn.io/ https://www.instantdb.com/ https://loro.dev/ https://electric-sql.com/ https://docs.y-sweet.dev/ https://syncedstore.org/docs/ https://collabs.readthedocs.io/en/latest/ https://remotestorage.io/ https://rxdb.i…

I wonder why there are so many, just people reinventing stuff that no one really needs? For me personally I have 4 of those as visited, pouchdb, automerge, loro and sqlsync of course. I was trying to fit such a tool into existing architectures that I deal with at work but nothing really makes sense. My guess is those solutions are in totally wrong abstraction layer, creators think that would be best thing since slice…

> I wonder why there are so many, just people reinventing stuff that no one really needs? Or were you being rhetorical?

What do people really need? Who defines and polices that?

> just reloading full state from the server is just easy to explain to the users and easy to implement

Is the green light on? If not, press the "power" button. Never underestimate the difficulty of explaining the simple to the uninterested.

The audience in this case is geeks like us, so it's probably ok to have wonky process until someone forks the project and fixes those problems.

Re: Stop syncing everything

#116

Post by [deleted] »

kreetx wrote: post #16
nilamo wrote: post #15

Earlier quoted context omitted.

I heavily disagree with the notion that most developers would rather query with something that isn't SQL.

Funny, as Triplit front page shows a query much like SQL: const deliveredMessagesQuery = client .query("messages") .Where("conversationId", "=", convoId) .Order("created_at", "DESC")

[deleted]

Re: Stop syncing everything

#117

Post by nilamo »

kreetx wrote: post #16
nilamo wrote: post #15

Earlier quoted context omitted.

I heavily disagree with the notion that most developers would rather query with something that isn't SQL.

Funny, as Triplit front page shows a query much like SQL: const deliveredMessagesQuery = client .query("messages") .Where("conversationId", "=", convoId) .Order("created_at", "DESC")

Exactly. Hard to improve on something so great.

We love a

``` select {what} from {where} left join {where} on {how} where {why} ```

A simple query, concisely answering every relevant question, while hiding all of the details of how any of it works. Beautiful.

Re: Stop syncing everything

#118

Post by kikimora »

bastawhiz wrote: post #103
ncruces wrote: post #74

Earlier quoted context omitted.

From the description, you can reapply transactions. How the system handles it (how much of it is up to the application, how much is handled in graft) I have no idea.

What does that mean though? How can you possibly reapply a failed transaction later? The database itself can't possibly know how to reconcile that (if it did, it wouldn't have been a failure in the first place). So it has to be done by the application, and that isn't always possible. There is still always the possibility of unavoidable data loss. "Consistency" is really easy, as it turns out, if you allow yourself to…

This! Solving merge conflicts in git is quite hard. Building an app such that it has a UI and use cases for merging every operation is just unrealistic. Perhaps if you limit yourself to certain domains like CRDTs or turn based games or data silos modified by only one customer it can be useful. I doubt it can work in general case.

Re: Stop syncing everything

#119

Post by upghost »

carlsverre wrote: post #68
upghost wrote: post #56

Man this looks super awesome. I will be extremely excited to ditch CouchDB for this (even tho I'm an Erlang fan). I'll certainly be keeping an eye on the project and I'll pitch in where I can!

Thank you!! From an extremely brief scan, it appears that Erlang wrappers around SQLite should be able to use the Graft SQLite extension just fine. Alternatively, it would be reasonably straight forward to wrap Graft Client (Rust library) directly in an Erlang NIF using something like https://github.com/rusterlium/rustler Let's make it happen! :)

Oh, I don't use anything about CouchDB except the REST API. sorry to send you on a wild goose chase >.Much appreciated, I'll keep you informed :)

Re: Stop syncing everything

#120

Post by bastawhiz »

kikimora wrote: post #118
bastawhiz wrote: post #103

Earlier quoted context omitted.

What does that mean though? How can you possibly reapply a failed transaction later? The database itself can't possibly know how to reconcile that (if it did, it wouldn't have been a failure in the first place). So it has to be done by the application, and that isn't always possible. There is still always the possibility of unavoidable data loss. "Consistency" is really easy, as it turns out, if you allow yourself to…

This! Solving merge conflicts in git is quite hard. Building an app such that it has a UI and use cases for merging every operation is just unrealistic. Perhaps if you limit yourself to certain domains like CRDTs or turn based games or data silos modified by only one customer it can be useful. I doubt it can work in general case.

The only situation I can think of where it's always safe is if the order that you apply changes to the state never matters:

- Each action increments or decrements a counter

- You have a log of timestamps of actions stored as a set

- etc.

If you can't model your changes to the data store as an unordered set of actions and have that materialize into state, you will have data loss.

Consider a scenario with three clients which each dispatch an action. If action 1 sets value X to true, action 2 sets it to true, and action 3 sets it to false, you have no way to know whether X should be true or false. Even with timestamps, unless you have a centralized writer you can't possibly know whether some/none/all of the timestamps that the clients used are accurate.

Truly a hard problem!

Post reply on HN