Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop syncing everything

sqlsync.dev

Post reply on HN
Print view View original

31–40 of 131 posts

Re: Stop syncing everything

#31

Post by tyre »

kiitos wrote: post #27
carlsverre wrote: post #26

Earlier quoted context omitted.

You're right - it's a bit confusing! I took a crack at explaining it in the blog post under the Consistency section: https://sqlsync.dev/posts/stop-syncing-everything/#consisten... The key idea is that if your system supports offline writes, then by definition the client making those writes can't have general purpose strict serializability. They have to exist under the assumption that when their transactions eventual…

What you've said here is totally different to what the repo docs claim. The guarantees of Graft's "commit" operation are properties of the Graft system itself. If commit is e.g. strict-serializable when clients satisfy one set of requirements, and isn't strict-serializable if clients don't satisfy those requirements, then "commit" is not strict-serializable.

The doc you linked and the author's response here do a good job of clarifying the conditions.

They're building an edge data store that has both local and global characteristics, with certain options meant for offline mode. It's reasonable to assume that the answer is more complicated when talking about the strict serializability of such a system.

Re: Stop syncing everything

#32

Post by feverzsj »

kiitos wrote: post #22

The consistency model doesn't seem to make sense. https://github.com/orbitinghail/graft/blob/main/docs/design.... > Graft clients commit locally and then asynchronously attempt to commit remotely. Because Graft enforces Strict Serializability globally, when two clients concurrently commit based on the same snapshot, one commit will succeed and the other will fail. OK, but, the API provides only a single commit operat…

It's basically single master asynchronous replication. And only works for sqlite's journal mode. The master saves all sqlite's journals as commit history and sends them to followers to replay them.

Re: Stop syncing everything

#33

Post by ccorcos »

How are permissions supposed to work? Suppose a page has data that I need to see and also has data I can’t see. Does this mean I need to demoralize my entire data model?

Re: Stop syncing everything

#34

Post by kiitos »

carlsverre wrote: post #29
kiitos wrote: post #27

Earlier quoted context omitted.

What you've said here is totally different to what the repo docs claim. The guarantees of Graft's "commit" operation are properties of the Graft system itself. If commit is e.g. strict-serializable when clients satisfy one set of requirements, and isn't strict-serializable if clients don't satisfy those requirements, then "commit" is not strict-serializable.

Just to make sure I understand correctly, would you agree that if clients always synchronously commit (i.e. wait until the MetaStore commit returns ok) before acknowledging any commit locally, the client will experience Strict Serializability? Assuming you agree with that, what would be a more clear way to explain the tradeoffs and resulting consistency models in the event that a client desires to asynchronously comm…

I think I'm probably operating with definitions of client and commit that are different than yours.

Specifically, I don't really see how a client can "commit locally" and "commit globally" as separate things. I understand a client to be something that interacts with your metastore API, which provides a single "commit" operation, that AFAICT will return success/failure based on local commit state, not global state.

Is that not correct?

Later on in the design.md you say

> The Client will be a Rust library ...

which might be the missing link in this discussion. Is the system model here assuming that clients are always gonna be Rust code in the same compilation unit as the Graft crate/library/etc.?

Re: Stop syncing everything

#36

Post by vineyardmike »

carlsverre wrote: post #28

Hey friends! Author of Graft here. Just want to say, huge thanks for all the great comments, stars, and support. Feels really nice to finally be building in public again. I'm going to force myself to sign off for the evening. Will be around first thing to answer any other questions that come up! I just arrived to Washington, DC to attend Antithesis BugBash[1] and if I don't get ahead of the jet lag I'm going to regre…

Thanks for sharing this, it looks really cool. I also wanted to explicitly mention that the graphics are great. IDK if you made them, or got help, but they do a great job explaining your point. It can be hard to create graphics for technical concepts like pages and databases, but these work well.

Re: Stop syncing everything

#37

Post by blatantly »

kreetx wrote: post #16
nilamo wrote: post #15

Earlier quoted context omitted.

I heavily disagree with the notion that most developers would rather query with something that isn't SQL.

Funny, as Triplit front page shows a query much like SQL: const deliveredMessagesQuery = client .query("messages") .Where("conversationId", "=", convoId) .Order("created_at", "DESC")

Well yeah choosing what to query, a filter and an order is at the heart of all query-like things. Buy SQL specifically is a text language not a fluent api.

Re: Stop syncing everything

#38

Post by chacham15 »

So, if I understand correctly, the consistency model is essentially git. I.e. you have a local copy, makes changes to it, and then when its time to "push" you can get a conflict where you can "rebase" or "merge".

The problem here is that there is no way to cleanly detect a conflict. The documentation talks about pages which have changed, but a page changing isnt a good indicator of conflict. A conflict can happen due to a read conflict. E.g.

Update Customer Id: "UPDATE Customers SET id='bar' WHERE id='foo'; UPDATE Orders SET customerId='bar' WHERE customerId='foo'"

Add Customer Purchase: "SELECT id FROM Customers WHERE email="blah"; INSERT INTO Orders(customerId, ...) VALUES("foo", ...);"

If the update task gets committed first and the pages for the Orders table are full (i.e. inserting causes a new page to allocated) these two operations dont have any page conflicts, but the result is incorrect.\

In order to fix this, you would need to track the pages read during the transaction in which the write occurred, but that could easily end up being the whole table if the update column isnt part of an index (and thus requiring a table scan).

Re: Stop syncing everything

#39

Post by londons_explore »

This approach has a problem when the mobile client is on the end of a very slow connection, yet the diff that needs to be synced is gigabytes, perhaps because it's an initial sync or the database got bloated for whatever reason.

A hybrid approach is to detect slow syncing (for example when sync hasn't completed after 5 seconds), and instead send queries directly to the server because there is a good chance the task the user wants to complete doesn't depend on the bloated records.

Re: Stop syncing everything

#40

Post by fulafel »

chacham15 wrote: post #38

So, if I understand correctly, the consistency model is essentially git. I.e. you have a local copy, makes changes to it, and then when its time to "push" you can get a conflict where you can "rebase" or "merge". The problem here is that there is no way to cleanly detect a conflict. The documentation talks about pages which have changed, but a page changing isnt a good indicator of conflict. A conflict can happen due…

In git the rebase of course isn't a sound operation either, the merge is heuristic and you're liable to get conflicts or silent mismerges.

Some simple examples: https://www.caktusgroup.com/blog/2018/03/19/when-clean-merge...

Post reply on HN