Does it needs to download whole database on startup, or can sync only what client queried?
Currently it's full db sync. Partial replication is in research.
Accidental database programming
61–70 of 310 posts
Re: Accidental database programming
#62This seems to be one of those problems that entirely disappears by ditching SPAs.
Using solutions from the Hotwire or htmx family would mean that a query is just a server query - making those fast is a better-understood problem.
Re: Accidental database programming
#63Don't give the user a mental model that reality can break ... badly, or invisibly I fear sync'ing databases instead of client server models is one of those - either your sync mechanism will just melt, or there are deep assumptions not met Inwoukd feel safer building a set of CRDT primitives to work with if I feel the need for fast UI and stick with forms submit for everything else -
I agree! One of my goals is to make the mental model of SQLSync easy to grok for the developers using it. I'm biased, but I find the rebase model much easier to understand than CRDTs.
Re: Accidental database programming
#64I’m familiar with this project - the creator is a friend. I’ll try to get him on here to answer questions. He’s a seasoned database architect. With SQLsync he’s made a way for frontend developers to query and update a remote database as if it was completely located right in the browser. Because it basically is. The power of WASM makes it possible to ship a whole SQLite database to the browser. The magic is in how it…
That sounds awfully like Couchbase, which allows you to query/update databases that will sync to remote and the back to peers. And you can control the process (auth/business logic) with sever side JavaScript plugin with ease.
Creator of Couchbase Mobile here — I’m doing a new web-based thing[1] with a similar reactive API. I’m hoping that my encrypted block replication makes it more of a “data anywhere” solution than a “local first” database. But the paradigm of powerful databases in the browser is definitely one I’m glad to see becoming popular.
Re: Accidental database programming
#65An old company I worked for used project management software with a check-in/out mechanism for making changes. When you "check out" a project it downloads a copy that you change locally, then "check in" uploads it back to the server. A project is "locked" while in the "checked out" state. We all felt it was an archaic mechanism in a word of live updating apps.
After 10 years of building SPA "web apps", that data synchronization mechanism feels ahead of its time.
Re: Accidental database programming
#66Give someone state and they'll have a bug one day, but teach them how to represent state in two separate locations that have to be kept in sync and they'll have bugs for a lifetime -ryg
Collollary: if you don't represent state in more than one place, you'll eventually run into loss of availability (accessibility), integrity, of existence of data Thus, bugs forever is a given.
Best to have one less major level of abstraction where that’s happening then.
Re: Accidental database programming
#67Earlier quoted context omitted.
History doesn't repeat itself, but it does rhyme. One upon a dark age, we had mainframes and dumb terminals. Then came the first age of the PC - let's pull everything to the client. Then came servers and "thin clients". With faster processors and cheaper storage came the second age of the PC, with only the permanent data storage left on the server. As the Internet grew, centralization came back: web services and the…
years and years ago on a C++ forum someone made an observation that was eerily similar to yours. I still remember it to this day as it stuck in my head. They made an observation that our industry goes in cyclical centralize/de-centralize cycles and that we we were (at the time) entering into a centralization cycle. Now here I am reading a comment that we're going back into a de-centralization cycle and I wouldn't be…
Could be - I've been making this observation for a long time. The cycles keep going. On the other hand, probably lots of other people have commented on it as well...
You may be right about the browser becoming the OS. Chromebooks were already a step in that direction. But JS/HTML/CSS really is a horrible combination for application programming. If the browser does become the OS, can we please get decent technology to work with?
Re: Accidental database programming
#68Offline/local-first based on SQLite seems hot right now. Third one I’m reading about this week. And it sounds good to me! But how does it compare to ElectricSQL[1] and PowerSync[2]? [1] https://electric-sql.com/ [2] https://powersync.com/
Indeed it's a very hot space! So exciting to see all the different approaches.
ElectricSQL and PowerSync are both tackling the very hard problem of partial replication. The idea is to build a general solution which allows a traditional centralized db to bidirectionally sync only what's needed on the client side - while still supporting optimistic mutations (and all the consistency/conflict stuff that goes along with that).
The downside is implementation complexity. Both require the ability to keep track of precisely the set of data on each client in order to push out changes to only that subset of the overall database. In addition, specifying which subsets of the database state to pull down requires a new DSL and is a new thing to learn (and optimize). That said, I'm stoked they are taking on this extremely hard problem so when SQLSync is ready for partial replication someone will have already figured out the best practices.
SQLSync, on the other hand, only supports full db sync. So every client will see a consistent view of the entire database. You might immediately wonder if this is a good idea - and for some apps, it's not. But consider a personal finance app. The main goal is cross device sync, cloud backup, offline capable, etc. In this case having the entire db stored on every device is probably what you want. Another example is a document oriented data model, such as Airtable. Each Airtable could be a distinct database, thus leaving it up to the client to manage which tables they care about.
(added in edit:) By focusing on full db sync, the sync engine is much simpler than solutions that support partial replication. One benefit of this is that the backend is very lightweight. Currently the demo (https://sqlsync-todo.pages.dev) runs entirely within Cloudflare Durable Objects using very little storage and CPU time.
SQLSync has a ton of work to do to make these use cases possible (still very much a prototype), but my initial tests have been extremely promising. Hope this helps!
(edit: clarified language regarding centralized dbs and full db sync. Also added paragraph regarding full db sync)
Re: Accidental database programming
#69This seems to be one of those problems that entirely disappears by ditching SPAs. Using solutions from the Hotwire or htmx family would mean that a query is just a server query - making those fast is a better-understood problem.
But, if I can be honest, solutions such as Hotwire or Livewire are not as snappy as a SPA.
I personally prefer InertiaJs [1], which is some kind of front-end router system with its state synced with the server in an "old style" fashion.
Re: Accidental database programming
#70An old company I worked for used project management software with a check-in/out mechanism for making changes. When you "check out" a project it downloads a copy that you change locally, then "check in" uploads it back to the server. A project is "locked" while in the "checked out" state. We all felt it was an archaic mechanism in a word of live updating apps. After 10 years of building SPA "web apps", that data sync…
Looks very similar to JEDI [0], an early Delphi VCS system that worked that way. It gave us the tranquility to know that no conflict would appear, as only one developer could work with a locked/checked out file at a time. There was no merge those days. In contrast, files that were frequently changed in every task would always cause a blocking between developers.