Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

91–100 of 131 posts

Re: Stop syncing everything

#91

Post by carlsverre »

carlsverre wrote: post #28

Hey friends! Author of Graft here. Just want to say, huge thanks for all the great comments, stars, and support. Feels really nice to finally be building in public again. I'm going to force myself to sign off for the evening. Will be around first thing to answer any other questions that come up! I just arrived to Washington, DC to attend Antithesis BugBash[1] and if I don't get ahead of the jet lag I'm going to regre…

Exciting to see all the continued energy around Graft today! If you want to continue the conversation feel free to join the Discord [1] or post a discussion on GitHub [2].

I'll be at Antithesis BugBash [3] for the next few days talking about Deterministic Simulation Testing (DST) with fellow DST nerds. If you're around please reach out so we can meet in person!

Either way, have an excellent day! :)

[1]: https://discord.gg/etFk2N9nzC

[2]: https://github.com/orbitinghail/graft

[3]: https://bugbash.antithesis.com/

Re: Stop syncing everything

#92

Post by stavros »

carlsverre wrote: post #90
stavros wrote: post #44

I didn't go into the implementation details so I won't comment on that, but I will say that this is a really important problem to solve. I've long wanted/needed an agnostic sync layer that apps could use to sync my changes across clients and servers, and it would be great if app developers didn't have to put any work into it, they'd just point their database to a URL and the database would take care of all the syncin…

Thank you! Glad we see the world the same way! I've also always wanted a general purpose sync layer that provided a simple consistency model to build on top of. And I made Graft open source to enable exactly that! Deploy it anywhere! Just let me know so I can better support your use case :)

Thanks for building SQLSync! You might like Klepmann's "local-first" talks, though I'm sure you're already familiar.

Re: Stop syncing everything

#93

Post by carlsverre »

canadiantim wrote: post #30
carlsverre wrote: post #12

Earlier quoted context omitted.

Thank you! Generally Turso has focused on operating more like a traditional network attached backend. Although that has changed recently with libsql and embedded replicas. I think at this point the main delta is they use traditional wal based physical replication while Graft is something new that permits trivial partial replication. Also, Graft is not exclusive to SQLite. It’s just transactional page addressed object…

Thanks! That's a good distinction Could you theoretically use it for e.g. DuckDB? (maybe not now, but with some work further down the line) What about a graph db like KuzuDB? or is it SQL only?

I think you could! Graft just provides transactions over sets of pages. As long as you can model your data in a compatible way it should work with Graft. A better question is how easy the system allows extensions to intercept filesystem operations. I'd prefer to avoid building a Fuse/NBD layer.

I've filed an issue for both:

https://github.com/orbitinghail/graft/issues/37

https://github.com/orbitinghail/graft/issues/38

Re: Stop syncing everything

#94

Post by NickM »

fauigerzigerk wrote: post #60
ncruces wrote: post #43

Earlier quoted context omitted.

They address this later on. If strict serializability is not possible, because your changes are based on a snapshot that is already invalid, you can either replay (your local transactions are not durable, but system-wide you regain serializability) or merge (degrading to snapshot isolation). As long as local unsynchronized transactions retain the page read set, and look for conflicts there, this should be sound.

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions. Dropping them all is technically consistent but it may be unsafe depending on the circumstances. E.g. a doc records an urgent referral but then the tx fails because admin staff has concurrently updated the patient's phone number or whatever. Automatically replaying is unsafe because c…

What I find hard to imagine is how the app should respond when synchronisation fails after locally committing a bunch of transactions... Manual merging may be the only safe option in many cases.

Yeah, exactly right. This is why CRDTs are popular: they give you well-defined semantics for automatic conflict resolution, and save you from having to implement all that stuff from scratch yourself.

The author writes that CRDTs "don’t generalize to arbitrary data." This is true, and sometimes it may be easier to your own custom app-specific conflict resolution logic than massaging your data to fit within preexisting CRDTs, but doing that is extremely tricky to get right.

It seems like the implied tradeoff being made by Graft is "you can just keep using the same data formats you're already using, and everything just works!" But the real tradeoff is that you're going to have to write a lot of tricky, error-prone conflict resolution logic. There's no such thing as a free lunch, unfortunately.

Re: Stop syncing everything

#95

Post by carlsverre »

stavros wrote: post #92
carlsverre wrote: post #90

Earlier quoted context omitted.

Thank you! Glad we see the world the same way! I've also always wanted a general purpose sync layer that provided a simple consistency model to build on top of. And I made Graft open source to enable exactly that! Deploy it anywhere! Just let me know so I can better support your use case :)

Thanks for building SQLSync! You might like Klepmann's "local-first" talks, though I'm sure you're already familiar.

You're welcome! And yea, I'm a huge fan of Klepmann and his local first content :) Thanks for the reminder!

Re: Stop syncing everything

#96

Post by ozim »

whizzter wrote: post #72
ozim wrote: post #59

Earlier quoted context omitted.

I wonder why there are so many, just people reinventing stuff that no one really needs? For me personally I have 4 of those as visited, pouchdb, automerge, loro and sqlsync of course. I was trying to fit such a tool into existing architectures that I deal with at work but nothing really makes sense. My guess is those solutions are in totally wrong abstraction layer, creators think that would be best thing since slice…

It's about the multiplayer application case, think Google Write/Sheets/etc. Applications with data that can change by multiple users and you can both see it live and the application (that keeps state in memory/localdb) is also resilient to disconnects. The reason people descend into this madness is because visible replication code is tricky and the general feeling is that it'll infect parts that shouldn't be infected…

I don’t see it you missed the context or I miss something.

Multiplayer documents are real time synchronized and since they are documents that’s totally not use case for DB synchronization.

All the tools are for offline to online data synchronization. Different use case than document.

Re: Stop syncing everything

#97

Post by bbrks »

whizzter wrote: post #72
ozim wrote: post #59

Earlier quoted context omitted.

I wonder why there are so many, just people reinventing stuff that no one really needs? For me personally I have 4 of those as visited, pouchdb, automerge, loro and sqlsync of course. I was trying to fit such a tool into existing architectures that I deal with at work but nothing really makes sense. My guess is those solutions are in totally wrong abstraction layer, creators think that would be best thing since slice…

It's about the multiplayer application case, think Google Write/Sheets/etc. Applications with data that can change by multiple users and you can both see it live and the application (that keeps state in memory/localdb) is also resilient to disconnects. The reason people descend into this madness is because visible replication code is tricky and the general feeling is that it'll infect parts that shouldn't be infected…

This gap is filled by the likes of Couchbase where a single org controls the majority of the stack (spoiler/disclaimer alert: I've been working on Couchbase's Sync for 8 years)

You get local the document-level atomicity for sync. Multi-document transaction support on server side, KV access, SQL inside JSON docs or even across multiple documents, Full Text Search, and fine-grained/RBAC for document-level synchronization - but the cost is as much lock-in as it is financial. You can't mix and match storage, query or sync without pretty big tradeoffs.

Re: Stop syncing everything

#98

Post by vvern »

eatonphil wrote: post #73

> [rqlite and dqlite] are focused on increasing SQLite’s durability and availability through consensus and traditional replication. They are designed to scale across a set of stateful nodes that maintain connectivity to one another. Little nitpick there, consensus anti-scales. You add more nodes and it gets slower. The rest of the section on rqlite and dqlite makes sense though, just not about "scale".

I’ll nitpick you back: if done correctly, consensus can have quite positive scaling consensus groups can have quite a positive impact on tail latency. As the membership size gets bigger, the expectation on the tail latency of the committing quorum goes down assuming independence and any sort of fat tailed distribution for individual participants.

Re: Stop syncing everything

#99

Post by ccorcos »

carlsverre wrote: post #88
ccorcos wrote: post #33

How are permissions supposed to work? Suppose a page has data that I need to see and also has data I can’t see. Does this mean I need to demoralize my entire data model?

There is simply so much to talk about here! Thanks for such an excellent question. First, a caveat: Graft currently has no permissions. Anyone with access to the Graft PageStore and MetaStore can read/write to any volume. This is obviously going to change - so I'll talk about what's planned rather than what exists. :) For writes, Graft can support fairly granular permission models. This is an advantage of handling wr…

Separate Volume per user makes sense... but to build an application where users can collaborate, I would need some way of fanning out writes to other users' databases. Any thoughts on how to do that in the context of Graft?

Re: Stop syncing everything

#100

Post by 0xbadcafebee »

I've always found these general solutions kind of confusing. All uses of distributed data are inherently flawed, and have specific limitations, so you have to pick a solution that closest matches your specific use case.

I like the idea behind graft, but it immediately runs into some complications. Like the fact that, as an edge device with unreliable networking, its client may not have availability to fetch the pages it needs when it needs them. If anything, what the client really needs is to fetch all the data whenever it can, so that when it does need to access it, it isn't waiting.

But if you have a lot of data, that could take forever! That's why the application needs to be more intelligent about what data it's requesting, and do things like create priorities and classes for its data so it can fetch what it needs the most first.

The fact that this is built for SQLite kind of reinforces the point. Distributed relational databases are usually a bad idea, because usually people want them to be ACID-like, but you can't enforce that in a distributed way. That's why things like CRDTs exist; you treat every "copy" as an independent thing, breaking the false narrative that you can really distribute the same thing in multiple places reliably. (And that shows you why relational databases just aren't good for replicating; to replicate an entire relational database with CRDTs, you need one CRDT per cell, which multiplies your data)

Post reply on HN