271–280 of 310 posts
Earlier quoted context omitted.
Those pesky backends are so annoying, so why don't we just put a backend on every client?
Schema and data migrations are too tricky, so why not have every client do it.
Many times the thought “what if we just shipped the database to the client” has crossed my mind in large multi tenant apps where individual datasets were relatively small. I’ve never gone far with it as it seems sufficiently outside the norm to be a cursed architectural pattern. Would be nice to find out I was wrong.
The correct answer is to move the UI back to the server, where the database already is, and just send html to the client side html renderer (or "web browser"). This whole post is just "front end and gone off the rails, over the cliff, and into the ocean."
Earlier quoted context omitted.
the real UX goal is to minimize data loss (ie. provide the ability to resume whatever the user was doing, from a point that's as recent as possible), and for this it becomes necessary to provide "auto-save" at that point there's already a diverging state problem, even if it's saved to localStorage (but obviously it's much better to save it on the server) it's absolutely ridiculous that we spent the last 10+ years sim…
> the real UX goal is to minimize data loss Can't say I've been asked to build optimistic updates for that reason, but the irony there is pretty rich if that's the UX goal you've seen it used for. In my experience optimistic rendering actually creates more data loss risk than it solves. Caching layers are involved, state is duplicated, and routing comes into question as users could leave the page before a failure sta…
For example the user wants to interact with the service, order something, check for new messages or send one.
On the happy path things are trivial, easy, synchronous, even quite resilient. There's not much complexity to manage on "no new message" or "add to cart, yes, buy with saved card, ship to saved address, k thx bye", if the browser or server crashes, or there's a network error, the interaction is short, easy to retry, problems are not hidden (quasi instant feedback)
But as the total interaction time increases, as the state the user would need to reproduce grows (adding new card, shipping/billing address, adding a lot of stuff into the card, writing a long message) the need for partial state sync arises.
There is an interaction here between the "what gets measured gets managed" principle and the sunk cost fallacy. The problem with databases is actually complexity. Any individual feature is more or less safe, but around the time reliability, caching and indexes get matched together there is a complexity explosion and it doesn't (normally, anyhow) make sense to implement a domain-specific DB (call is a DSD?). But, arou…
What is exactly hard about sql? Every dev imho should know it. And sql syntax is good and proven too be long lasting. Maybe investing some time in actually learning it instead of bashing it will help you further.
Earlier quoted context omitted.
Reminds me of Meteorjs. It would let you sync a subset of your data to the client and then the client could query it any which way it wanted. They called this “Minimongo”.
I've used Meteor. I thought it was a good system. It didn't have offline capability, at least not back when I used it. It really needed to be connected to work. But conceptually, yes, it had a very similar system.
I remember having problems with 200 users serving from my above average dev pc for testing internal tools.
It's a DX dream though.
Earlier quoted context omitted.
Tbf, the WebSQL standard was not well-written from how I've heard that story told. It was bug-for-bug exactly standardized to a particular version of SQLite, which is not a good way to write a standard.
The important thing is - Firefox has been slowly dying for a decade and SQLite has taken over the world.
Coming from the backend world, I've only done small frontends that don't get this complicated. Assuming there are legit reasons for a FE to have complex state, a relational database is probably one of the first things you need. It's almost an automatic decision on a backend to have one, same should apply here. Using SQLite instead of Redux sounds reasonable.
> Using SQLite instead of Redux sounds reasonable. The trick is reactivity. `redux` handles surgically updating the view based state changes. It's not enough to have sqlite, you also need reactivity. Further, apply strict types on top of sqlite/SQL is another big challenge.
Earlier quoted context omitted.
How is this approach meant to handle data visibility and access control? Often a large part of a backend is materializing raw data into a form that the active user is allowed to view.
So if the user owns all their own data, their "data view" is their data set. A To-Do system, a personal finance app, any kind of note-taking or personal record keeping fits this model. You create a database per user and the auth and sync are all self contained within that database. This system is multi-master, which means that any change on a client or on the server will be replicated to every other. There is no "aut…
Sorry for perhaps reducing what SQLSync does to any over simplistic description, but is it effectively applying the CRDT pattern on top of the client (SQLSync/sqlite) and servers (Some OLTP) individual operation logs? edit: I mean rebase, not CRDT
Basically each client maintains their own log of mutations. These mutations are applied optimistically to the local instance of SQLite, and in parallel we replicate the log to the server.
On the server side, it reads from all of the individual client logs in a deterministic order and applies the mutations to it's own SQLite database. Under the hood SQLSync hijacks all of the writes to storage and organises them into a format that's easy to replicate.
Finally, the the storage log from the server is replicated back down to each of the clients. This leaves the clients in a weird position as they have two versions of the database which may have diverged (probably). So to finish this up, the clients throw away their local changes and reset to the server state (i.e. git reset --hard). Then since there may be some mutations that were executed on the client but not yet run on the server the client simply re-runs those mutations (i.e. git rebase).
In this way the system continuously keeps itself in sync with the server and other clients.
Conflicts are handled by logic in the reducer which is able to inspect the state of the database to figure out what to do. Yes, this does require writing the reducer code carefully - however in testing I've found this not to be too bad because:
1. A lot of SQL operations automatically converge pretty nicely (and you probably already have to think about them converging in your existing REST api's or w/e backend api you are writing). Think about patterns like `insert...on conflict do...` for example.
2. Since the reducer is isolated and comes with a type description of all possible mutations, it's very easy to unit test the reducer with different orderings. Basically think of it as deterministic simulation testing for your API. Something that's pretty hard to do with normal backend architectures without a lot of mocking and architecture.
Hopefully that helps!
Earlier quoted context omitted.
I agree! One of my goals is to make the mental model of SQLSync easy to grok for the developers using it. I'm biased, but I find the rebase model much easier to understand than CRDTs.
I feel like you might be miss the point of the parent comment. Synchronizing between databases is a notoriously difficult problem. It's really hard to do while avoiding race conditions. To be fair, I don't understand the "git rebase" technique your project uses, but I'm doubtful it solves these problems. The underlying issue is that users of SQLsync are going to assume that consistency works, but in suble and unsuspe…
Also, I'm considering adding a CRDT column type to SQLite to enable embedded collaborative use cases while using the relational model for the rest of your data's structure.