Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

81–90 of 131 posts

Re: Stop syncing everything

#81

Post by carlsverre »

imagio wrote: post #71
carlsverre wrote: post #64

Earlier quoted context omitted.

Glad you enjoyed Graft! The system your describing sounds very cool! It's actually quite similar to SQLSync. The best description of how SQLSync represents and replays intentions (SQLSync calls them mutations) is this talk I did at WasmCon 2023: https://www.youtube.com/watch?v=oLYda9jmNpk

Cool! That's an interesting approach putting the actions in wasm. I'm going for something more tightly integrated into an application rather than entirely in the database layer. The actions in my prototype are just TS functions (actually Effects https://effect.website/ but same idea) that can arbitrarily read and write to the client local database. This does put some restrictions on the app -- it has to define all mu…

Woah that's awesome. Using Effect to represent mutations is brilliant!

Any chance your project is public? I'd love to dig into the details. Alternatively would you be willing to nerd out on this over a call sometime? You can reach me at hello [at] orbitinghail [dotdev]

Re: Stop syncing everything

#82

Post by carlsverre »

theSherwood wrote: post #42

This is a very interesting approach. Using pages as the basic sync unit seems to simplify a lot. It also makes the sync of arbitrary bytes possible. But it does seem that if your sync is this coarse-grained that there would be lots of conflicting writes in applications with a lot of concurrent users (even if they are updating semantically unrelated data). Seems like OT or CRDT would be better in such a use-case. I'd…

Thank you! You're 1000% correct, Graft's approach is not a good fit for high write contention on a single Volume. Graft instead is designed for architectures that can either partition writes[^1] or can represent writes as bundled mutations and apply them in a single location that can enforce order (the SQLSync model).

Basically, Graft is not the full story, but as you point out - because it's so simple it's easy to build different solutions on top of it.

[^1]: either across Volumes or across pages, Graft can handle automatically merging non-overlapping page sets

Re: Stop syncing everything

#85

Post by mdavidn »

kiitos wrote: post #34
carlsverre wrote: post #29

Earlier quoted context omitted.

Just to make sure I understand correctly, would you agree that if clients always synchronously commit (i.e. wait until the MetaStore commit returns ok) before acknowledging any commit locally, the client will experience Strict Serializability? Assuming you agree with that, what would be a more clear way to explain the tradeoffs and resulting consistency models in the event that a client desires to asynchronously comm…

I think I'm probably operating with definitions of client and commit that are different than yours. Specifically, I don't really see how a client can "commit locally" and "commit globally" as separate things. I understand a client to be something that interacts with your metastore API, which provides a single "commit" operation, that AFAICT will return success/failure based on local commit state, not global state. Is…

Graft's definition sounds more like a Git "commit" than one found in a SQL standard. Perhaps that's the source of this confusion?

Re: Stop syncing everything

#86

Post by imagio »

carlsverre wrote: post #81
imagio wrote: post #71

Earlier quoted context omitted.

Cool! That's an interesting approach putting the actions in wasm. I'm going for something more tightly integrated into an application rather than entirely in the database layer. The actions in my prototype are just TS functions (actually Effects https://effect.website/ but same idea) that can arbitrarily read and write to the client local database. This does put some restrictions on the app -- it has to define all mu…

Woah that's awesome. Using Effect to represent mutations is brilliant! Any chance your project is public? I'd love to dig into the details. Alternatively would you be willing to nerd out on this over a call sometime? You can reach me at hello [at] orbitinghail [dotdev]

I haven't put it up in a github repo yet, it's not finished enough, but I will after spending a little more time hacking on it. I'll try to remember to come back here to comment when I do =)

Using effect really doesn't introduce anything special -- plain async typescript functions would work fine too. My approach is certainly a lot less sophisticated than yours with Graft or SQLSync! I just like using Effect. It makes working with typescript a lot more pleasant.

Re: Stop syncing everything

#88

Post by carlsverre »

ccorcos wrote: post #33

How are permissions supposed to work? Suppose a page has data that I need to see and also has data I can’t see. Does this mean I need to demoralize my entire data model?

There is simply so much to talk about here! Thanks for such an excellent question.

First, a caveat: Graft currently has no permissions. Anyone with access to the Graft PageStore and MetaStore can read/write to any volume. This is obviously going to change - so I'll talk about what's planned rather than what exists. :)

For writes, Graft can support fairly granular permission models. This is an advantage of handling writes in the PageStore. Depending on the data being stored in a Volume, a future PageStore version could reject writes based on inspecting the uploaded pages. This would increase the load on the PageStore, but since it's designed to run on the edge and horizontally scale like crazy (stateless) it seems like it would work.

Reads, on the other hand, are a lot more tricky. The simplest approach is to partition data across Volumes such that you can enforce read permissions at the Volume level. This isn't a great solution and will certainly limit the kinds of workloads that are well aligned with Graft. A more complex approach is to layer Volumes. Effectively virtualizing a single database that internally writes rows to different layers depending on access permissions. This second approach offers a slightly nicer user experience, at the cost of complexity and query performance.

For now though, Graft is best suited to workloads that can partition data and permissions across Volumes.

Re: Stop syncing everything

#89

Post by carlsverre »

HugoMoran wrote: post #66

Really like the look of this. Any thoughts on how it could be extended to work with duckdb?

Thanks! And great idea. I haven't deep dived into DuckDB's storage format yet, however I know that they split up the data into row groups which internally are columnar. I'm not sure how that maps to the pages in the file. If it doesn't align well, or they do a lot of shifting style writes (writes that cause all subsequent data to shift forward or backward) it may not be a great fit for Graft.

Either way, it's 100% a great idea that I'd like to explore. If any DuckDB contributors are reading I'd love to know if it would work!

Re: Stop syncing everything

#90

Post by carlsverre »

stavros wrote: post #44

I didn't go into the implementation details so I won't comment on that, but I will say that this is a really important problem to solve. I've long wanted/needed an agnostic sync layer that apps could use to sync my changes across clients and servers, and it would be great if app developers didn't have to put any work into it, they'd just point their database to a URL and the database would take care of all the syncin…

Thank you! Glad we see the world the same way! I've also always wanted a general purpose sync layer that provided a simple consistency model to build on top of.

And I made Graft open source to enable exactly that! Deploy it anywhere! Just let me know so I can better support your use case :)

Post reply on HN