Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Stop syncing everything

#121

Post by aboodman »

carlsverre wrote: post #24
conradev wrote: post #5

> After a client pulls a graft, it knows exactly what’s changed. It can use that information to determine precisely which pages are still valid and which pages need to be fetched Curious how this compares to Cloud-Backed SQLite’s manifest: https://sqlite.org/cloudsqlite/doc/trunk/www/index.wiki It’s similar to your design (sending changed pages), but doesn’t need any compute on the server, which I think is a huge win…

Thanks for bringing that up! Cloud-Backed SQLite (CBS) is an awesome project and perhaps even more importantly a lot more mature than Graft. But here is my overview of what's different: CBS uses manifests and blocks as you point out. This allows readers to pull a manifest and know which blocks can be reused and which need to be pulled. So from that perspective it's very similar. The write layer is pretty different, m…

When you say “rebase” what do you mean exactly? Is it the application replaying the transaction including the interactive code, or is it the database writing the same rows/pages that were written last time?

Re: Stop syncing everything

#122

Post by vladstudio »

isaachinman wrote: post #113
vladstudio wrote: post #57

in case anyone finds this useful, here's the slowly growing collection of links to similar tools: https://tinybase.org/ https://www.evolu.dev/ https://replicache.dev/ https://fireproof.storage/ https://vlcn.io/ https://www.instantdb.com/ https://loro.dev/ https://electric-sql.com/ https://docs.y-sweet.dev/ https://syncedstore.org/docs/ https://collabs.readthedocs.io/en/latest/ https://remotestorage.io/ https://rxdb.i…

Wrote a very-related blog post here: https://marcoapp.io/blog/offline-first-landscape

thanks for the link! just in time for me.

Re: Stop syncing everything

#123

Post by whizzter »

ozim wrote: post #96
whizzter wrote: post #72

Earlier quoted context omitted.

It's about the multiplayer application case, think Google Write/Sheets/etc. Applications with data that can change by multiple users and you can both see it live and the application (that keeps state in memory/localdb) is also resilient to disconnects. The reason people descend into this madness is because visible replication code is tricky and the general feeling is that it'll infect parts that shouldn't be infected…

I don’t see it you missed the context or I miss something. Multiplayer documents are real time synchronized and since they are documents that’s totally not use case for DB synchronization. All the tools are for offline to online data synchronization. Different use case than document.

Not really, the fundamental problem shared for all the tools mentioned by GP is the same of concurrent edits. We as an industry typically ignore it in regular CRUD settings due to low conflicting operation frequency, but it gets problematic in offline and multiplayer settings since those cause it to become more common/visible.

As for documents vs DB, technically you can often compose documents into a number of tables. And having it decomposed into a regular DB simplifies a lot of visibility tasks since you don't need to built an extra layer to manage that and/or you might want to implement partial permissions into something that feels "document like" but really pertains to a larger system.

Re: Stop syncing everything

#124

Post by imagio »

carlsverre wrote: post #81
imagio wrote: post #71

Earlier quoted context omitted.

Cool! That's an interesting approach putting the actions in wasm. I'm going for something more tightly integrated into an application rather than entirely in the database layer. The actions in my prototype are just TS functions (actually Effects https://effect.website/ but same idea) that can arbitrarily read and write to the client local database. This does put some restrictions on the app -- it has to define all mu…

Woah that's awesome. Using Effect to represent mutations is brilliant! Any chance your project is public? I'd love to dig into the details. Alternatively would you be willing to nerd out on this over a call sometime? You can reach me at hello [at] orbitinghail [dotdev]

As promised, here's the code. I decided to call it Synchrotron. Not ready for use yet but there is design/planning in the readme and a suite of passing tests showing that the approach can work. https://github.com/evelant/synchrotron

Re: Stop syncing everything

#125

Post by fatherzine »

fauigerzigerk wrote: post #105
NickM wrote: post #94

Earlier quoted context omitted.

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions... Manual merging may be the only safe option in many cases. Yeah, exactly right. This is why CRDTs are popular: they give you well-defined semantics for automatic conflict resolution, and save you from having to implement all that stuff from scratch yourself. The author writes that CR…

The problem I have with CRDTs is that while being conflict-free in a technical sense they don't allow me to express application level constraints. E.g, how do you make sure that a hotel room cannot be booked by more than one person at a time or at least flag this situation as a constraint violation that needs manual intervention? It's really hard to get anywhere close to the universal usefulness and simplicity of cen…

"How do you make sure that a hotel room cannot be booked by more than one person at a time" Excellent question! You don't. Instead, assuming a globally consistent transaction ordering, eg Spanner's TrueTime, but any uuid scheme suffices, it becomes a tradeoff between reconciliation latency and perceived unreliability. A room may be booked by several persons at a time, but eventually only one of them will win the reconciliation process.

    A: T.uuid3712[X] = reserve X
    ...
    B: T.uuid6214[X] = reserve X  // eventually loses to A because of uuid ordering
    ...
    A
The engineering challenge becomes to reduce the reconciliation latency window to something tolerable to users. If the reconciliation latency is small enough, then a blocking API can completely hide the unreliability from users.

Re: Stop syncing everything

#126

Post by carlsverre »

imagio wrote: post #124
carlsverre wrote: post #81

Earlier quoted context omitted.

Woah that's awesome. Using Effect to represent mutations is brilliant! Any chance your project is public? I'd love to dig into the details. Alternatively would you be willing to nerd out on this over a call sometime? You can reach me at hello [at] orbitinghail [dotdev]

As promised, here's the code. I decided to call it Synchrotron. Not ready for use yet but there is design/planning in the readme and a suite of passing tests showing that the approach can work. https://github.com/evelant/synchrotron

Sweet!! Will check it out shortly. Thanks for sharing :)

Re: Stop syncing everything

#127

Post by kiitos »

mdavidn wrote: post #85
kiitos wrote: post #34

Earlier quoted context omitted.

I think I'm probably operating with definitions of client and commit that are different than yours. Specifically, I don't really see how a client can "commit locally" and "commit globally" as separate things. I understand a client to be something that interacts with your metastore API, which provides a single "commit" operation, that AFAICT will return success/failure based on local commit state, not global state. Is…

Graft's definition sounds more like a Git "commit" than one found in a SQL standard. Perhaps that's the source of this confusion?

That may be the case, but the Git notion of a "commit" is a domain concept that is unrelated to, and incompatible with, the distsys notion of a "commit". And terminology like "strict serialization" and "snapshot isolation" and so on -- which this system and its docs reference -- are distsys concepts, not Git concepts. So "commit" needs to abide distsys rules, not Git rules, as you can't really mix-and-match definitions across domains.

Re: Stop syncing everything

#128

Post by ccorcos »

carlsverre wrote: post #101
ccorcos wrote: post #99

Earlier quoted context omitted.

Separate Volume per user makes sense... but to build an application where users can collaborate, I would need some way of fanning out writes to other users' databases. Any thoughts on how to do that in the context of Graft?

If you're doing volume per user, but also want to do cross-user collab you might want to change the model slightly. Rather than one volume per user, consider one volume per "access unit". For example a document or group could be a volume. As an example, let's say your building something like Google Sheets on top of Graft. Each document would be an independent Volume. This matches how Sharing works in Google Sheets, a…

Let's consider something like Notion or Slack. It's sort of document-based but not entirely -- you can mention other users and other channels names. Some users have access to some channels and not others. And in Notion's case, users may have unique access to 1000s of different documents.

It seems like your solution requires essentially sharding your data based on permission which can get pretty complicated for many collaboration-based apps with lots of shared content and granular permission controls.

Re: Stop syncing everything

#129

Post by kreetx »

nilamo wrote: post #117
kreetx wrote: post #16

Earlier quoted context omitted.

Funny, as Triplit front page shows a query much like SQL: const deliveredMessagesQuery = client .query("messages") .Where("conversationId", "=", convoId) .Order("created_at", "DESC")

Exactly. Hard to improve on something so great. We love a ``` select {what} from {where} left join {where} on {how} where {why} ``` A simple query, concisely answering every relevant question, while hiding all of the details of how any of it works. Beautiful.

SQL is a relational algebra language which, unlike imperative languages, is compiled into an execution plan depending on the existence of indexes, table sizes, data types, etc. The reason you should love it is that you don't need to know the details when the data model is set up well. And DBMSs also have commands to view the execution plan, should it have otherwise been slow, such that you know what to do to improve it. It's really not that complicated once you learn how it works.

Re: Stop syncing everything

#130

Post by conradev »

aboodman wrote: post #121
carlsverre wrote: post #24

Earlier quoted context omitted.

Thanks for bringing that up! Cloud-Backed SQLite (CBS) is an awesome project and perhaps even more importantly a lot more mature than Graft. But here is my overview of what's different: CBS uses manifests and blocks as you point out. This allows readers to pull a manifest and know which blocks can be reused and which need to be pulled. So from that perspective it's very similar. The write layer is pretty different, m…

When you say “rebase” what do you mean exactly? Is it the application replaying the transaction including the interactive code, or is it the database writing the same rows/pages that were written last time?

SQLite has an experimental change, "BEGIN CONCURRENT", which uses optimistic locking to allow concurrent non-overlapping writes: https://www.sqlite.org/src/doc/begin-concurrent/doc/begin_co...

So the "rebase" could be the exact same pages (no compute) if none of the writes overlap. If they do, you'd likely need to replay the transaction and make new pages (some compute).

Post reply on HN