Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

101–110 of 131 posts

Re: Stop syncing everything

#101

Post by carlsverre »

ccorcos wrote: post #99
carlsverre wrote: post #88

Earlier quoted context omitted.

There is simply so much to talk about here! Thanks for such an excellent question. First, a caveat: Graft currently has no permissions. Anyone with access to the Graft PageStore and MetaStore can read/write to any volume. This is obviously going to change - so I'll talk about what's planned rather than what exists. :) For writes, Graft can support fairly granular permission models. This is an advantage of handling wr…

Separate Volume per user makes sense... but to build an application where users can collaborate, I would need some way of fanning out writes to other users' databases. Any thoughts on how to do that in the context of Graft?

If you're doing volume per user, but also want to do cross-user collab you might want to change the model slightly. Rather than one volume per user, consider one volume per "access unit". For example a document or group could be a volume.

As an example, let's say your building something like Google Sheets on top of Graft. Each document would be an independent Volume. This matches how Sharing works in Google Sheets, as each user added to the Volume could either be granted read or write permissions to the entire sheet.

Re: Stop syncing everything

#102

Post by codeulike »

vladstudio wrote: post #57

in case anyone finds this useful, here's the slowly growing collection of links to similar tools: https://tinybase.org/ https://www.evolu.dev/ https://replicache.dev/ https://fireproof.storage/ https://vlcn.io/ https://www.instantdb.com/ https://loro.dev/ https://electric-sql.com/ https://docs.y-sweet.dev/ https://syncedstore.org/docs/ https://collabs.readthedocs.io/en/latest/ https://remotestorage.io/ https://rxdb.i…

"nearly every problem these days is a synchronisation problem"

https://news.ycombinator.com/item?id=43434239

Re: Stop syncing everything

#103

Post by bastawhiz »

ncruces wrote: post #74
bastawhiz wrote: post #69

Earlier quoted context omitted.

> your local transactions are not durable This manifests itself to the user as just data loss, though. You do something, it looks like it worked, but then it goes away later.

From the description, you can reapply transactions. How the system handles it (how much of it is up to the application, how much is handled in graft) I have no idea.

What does that mean though? How can you possibly reapply a failed transaction later? The database itself can't possibly know how to reconcile that (if it did, it wouldn't have been a failure in the first place). So it has to be done by the application, and that isn't always possible. There is still always the possibility of unavoidable data loss.

"Consistency" is really easy, as it turns out, if you allow yourself to simply drop any inconvenient transactions at some arbitrary point in the future.

Re: Stop syncing everything

#104

Post by conradev »

carlsverre wrote: post #24
conradev wrote: post #5

> After a client pulls a graft, it knows exactly what’s changed. It can use that information to determine precisely which pages are still valid and which pages need to be fetched Curious how this compares to Cloud-Backed SQLite’s manifest: https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki It’s similar to your design (sending changed pages), but doesn’t need any compute on the server, which I think is a huge win…

Thanks for bringing that up! Cloud-Backed SQLite (CBS) is an awesome project and perhaps even more importantly a lot more mature than Graft. But here is my overview of what's different: CBS uses manifests and blocks as you point out. This allows readers to pull a manifest and know which blocks can be reused and which need to be pulled. So from that perspective it's very similar. The write layer is pretty different, m…

That makes total sense! Beyond the efficiency of no servers, the other aspect of CBS that is appealing to me is its simplicity.

Graft introduces a lot of new concepts, and while they might be necessary to achieve the more ambitious goals, it feels a like a lot. Running an HTTP API with Protobuf is a lot of API surface area to maintain and evolve going forward.

Write concurrency for CBS is "one writer per bucket", which is usable in a "one bucket per user" configuration. You can mediate a client's blob storage access with signed URLs. It's not great, though, and you have to roll your own conflict resolution.

The most interesting choice to me here is to handle conflict resolution (rebasing/forking/resetting) at the storage engine (page) level. For non-overlapping writes, I can see how rebasing is quite effective.

Re: Stop syncing everything

#105

Post by fauigerzigerk »

NickM wrote: post #94
fauigerzigerk wrote: post #60

Earlier quoted context omitted.

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions. Dropping them all is technically consistent but it may be unsafe depending on the circumstances. E.g. a doc records an urgent referral but then the tx fails because admin staff has concurrently updated the patient's phone number or whatever. Automatically replaying is unsafe because c…

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions... Manual merging may be the only safe option in many cases. Yeah, exactly right. This is why CRDTs are popular: they give you well-defined semantics for automatic conflict resolution, and save you from having to implement all that stuff from scratch yourself. The author writes that CR…

The problem I have with CRDTs is that while being conflict-free in a technical sense they don't allow me to express application level constraints.

E.g, how do you make sure that a hotel room cannot be booked by more than one person at a time or at least flag this situation as a constraint violation that needs manual intervention?

It's really hard to get anywhere close to the universal usefulness and simplicity of centralised transactions.

Re: Stop syncing everything

#106

Post by gejose »

carlsverre wrote: post #28

Hey friends! Author of Graft here. Just want to say, huge thanks for all the great comments, stars, and support. Feels really nice to finally be building in public again. I'm going to force myself to sign off for the evening. Will be around first thing to answer any other questions that come up! I just arrived to Washington, DC to attend Antithesis BugBash[1] and if I don't get ahead of the jet lag I'm going to regre…

Great work here! A bit of a silly question - but can I ask you what tool you used to build the beautiful diagrams on the page (eg: https://sqlsync.dev/_astro/pull_changes.DjOYfgBf_2biXxv.webp)

Re: Stop syncing everything

#107

Post by carlsverre »

gejose wrote: post #106
carlsverre wrote: post #28

Hey friends! Author of Graft here. Just want to say, huge thanks for all the great comments, stars, and support. Feels really nice to finally be building in public again. I'm going to force myself to sign off for the evening. Will be around first thing to answer any other questions that come up! I just arrived to Washington, DC to attend Antithesis BugBash[1] and if I don't get ahead of the jet lag I'm going to regre…

Great work here! A bit of a silly question - but can I ask you what tool you used to build the beautiful diagrams on the page (eg: https://sqlsync.dev/_astro/pull_changes.DjOYfgBf_2biXxv.webp )

Thank you! I made all the diagrams in https://excalidraw.com/. Been using it for years and absolutely love it!

Re: Stop syncing everything

#108

Post by carlsverre »

conradev wrote: post #104
carlsverre wrote: post #24

Earlier quoted context omitted.

Thanks for bringing that up! Cloud-Backed SQLite (CBS) is an awesome project and perhaps even more importantly a lot more mature than Graft. But here is my overview of what's different: CBS uses manifests and blocks as you point out. This allows readers to pull a manifest and know which blocks can be reused and which need to be pulled. So from that perspective it's very similar. The write layer is pretty different, m…

That makes total sense! Beyond the efficiency of no servers, the other aspect of CBS that is appealing to me is its simplicity. Graft introduces a lot of new concepts, and while they might be necessary to achieve the more ambitious goals, it feels a like a lot. Running an HTTP API with Protobuf is a lot of API surface area to maintain and evolve going forward. Write concurrency for CBS is "one writer per bucket", whi…

Fair! CBS is absolutely a killer solution to the same class of problems.

Re: Stop syncing everything

#110

Post by NickM »

fauigerzigerk wrote: post #105
NickM wrote: post #94

Earlier quoted context omitted.

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions... Manual merging may be the only safe option in many cases. Yeah, exactly right. This is why CRDTs are popular: they give you well-defined semantics for automatic conflict resolution, and save you from having to implement all that stuff from scratch yourself. The author writes that CR…

The problem I have with CRDTs is that while being conflict-free in a technical sense they don't allow me to express application level constraints. E.g, how do you make sure that a hotel room cannot be booked by more than one person at a time or at least flag this situation as a constraint violation that needs manual intervention? It's really hard to get anywhere close to the universal usefulness and simplicity of cen…

Yeah, this is a limitation, but generally if you have hard constraints like that to maintain, then yeah you probably should be using some sort of centralized transactional system to avoid e.g. booking the same hotel room to multiple people in the first place. Even with perfect conflict resolution, you don't want to tell someone their booking is confirmed and then later have to say "oh, sorry, never mind, somebody else booked that room and we just didn't check to verify that at the time."

But this isn't a problem specific to CRDTs, it's a limitation with any database that favors availability over consistency. And there are use cases that don't require these kinds of constraints where these limitations are more manageable.

Post reply on HN