Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accidental database programming

sqlsync.dev

Post reply on HN
Print view View original

21–30 of 310 posts

Re: Accidental database programming

#21

Post by neilk »

I’m familiar with this project - the creator is a friend. I’ll try to get him on here to answer questions.

He’s a seasoned database architect. With SQLsync he’s made a way for frontend developers to query and update a remote database as if it was completely located right in the browser. Because it basically is. The power of WASM makes it possible to ship a whole SQLite database to the browser. The magic is in how it syncs from multiple clients with a clever but simple reactive algorithm.

It’s a radical attack on the whole problem. Much of our work as developers is about syncing data. When you start looking at React and REST APIs as a kind of sync procedure, this approach can open a lot of new possibilities. You don’t have to write a weird bespoke database of trees of objects fetched and cached from the API any more. You can just update and query it locally, with all the power of a relational database.

Re: Accidental database programming

#22

Post by carlsverre »

athrowaway3z wrote: post #19

This looks interesting and i might give it a try, but after watching the talk i'm still a bit unclear why you choose for wasm-in-wasm for the reducer. I suspect you would be better off by creating a rust reducer trait, and lifting that wasm-in-wasm complexity into a new crate implementing the reducer trait through wasm-in-wasm for the people who want that. But maybe i'm missing something.

Totally fair question. Nothing is set in stone - but this approach made it very easy for me to run precisely the same code on both a generic backend (CloudFlare Durable Objects) and in the frontend stack.

As for wasm-in-wasm specifically. It would be nice to experiment with the component model to load reducers alongside SQLSync - but the UX isn't quite there yet.

Re: Accidental database programming

#23

Post by carlsverre »

pqdbr wrote: post #9

Can someone explain me how it's syncing the state between two different devices without any activity in the Network tab in DevTools, not even WS traffic? I get that you can sync state between browser tabs, but I'm trying on two different devices (iPhone and Desktop). And as far as I can tell, the Wasm layer can't perform network requests directly. UPDATE: In the console tab I can see 'coordinatorUrl: 'wss://sqlsync.o…

Good catch! SQLSync runs in a shared worker to enable cross-tab reactivity and centralise both local storage and the replication system. You can inspect the worker at the magic url: chrome://inspect/#workers

Re: Accidental database programming

#24

Post by _heimdall »

threatofrain wrote: post #17
bob1029 wrote: post #16

Trying to synchronize state between client & server is a cursed problem. You can sidestep it altogether if you make mild UX sacrifices and revert to something more akin to the PHP/SSR model. SPA is nice, but multipart form posts still work. Just the tiniest amount of javascript can smooth out most of the remaining rough edges. Our latest web products utilize the following client-side state: 3rd party IdP claims for a…

It's a very common trend for consumer-facing GUI's to have optimistic rendering, and if you're doing that then you're juggling client/server state. I still see spinning loaders here and then but they're generally for initial content load; e.g., does Gmail make you wait when you archive an email?

Not the GP, but I would include optimistic rendering on the list of common patterns that really are a bad idea.

Optimistic rendering means your frontend and backend are tightly coupled, error recovery and synchronization is much more complex, and you are locked into (likely heavy) frontend rendering patterns that add even more complexity and coupling.

We've spent well over a decade trying to avoid the fact that frontend actions require backend logic to complete. Its a losing battle that's just made worse by trying to paper over it.

Edit to clarify: this isn't a direct comment on the OP tool. I haven't used this tool directly but having a hunch it does solve some important use cases for common patterns.

Re: Accidental database programming

#25

Post by carlsverre »

jokethrowaway wrote: post #14
yewenjie wrote: post #3

Is this supposed to be run on the server? Then how does it really solve the frontend side of issues, I'm just trying to understand.

Not involved with the project but - this is a database which run client side and sync with a database on the server

Precisely! The same database (SQLite) is running on both the client and the server. SQLSync provides a custom storage layer to SQLite that keeps everything in sync. As changes are made locally (optimistically) they are synced into the cloud where they are eventually applied to the primary db replica. The clients subscribe to changes from the primary db and then rebase themselves to stay in sync.

I really need to write up a detailed overview of how this works! Thanks for the feedback!

Re: Accidental database programming

#26

Post by carlsverre »

recusive_story wrote: post #13

Local to a webpage, do you feel building wrapper over indexedDB instead of sqlite would be better idea?

That's a great way to accomplish local storage, but requires a bit of gymnastics to build sync. By controlling the database entirely, SQLSync can provide very ergonomic sync to the developer and performance for the user.

So it's not that one is better than the other. Just the capabilities, performance, and test-ability differs.

Re: Accidental database programming

#27

Post by carlsverre »

tootie wrote: post #11

The thing I've used for this kind of problem is fusejs which is a lightweight search index. You can load it with a list of JSON documents and do structured or fuzzy string searches. I find it pretty well-suited to the kind of frontend experiences I need a lot of data for.

This is cool! Thanks for sharing. Sounds like Fuse would be a great solution for a read-only index. But what if you want to collaborate on the data with other people?

FWIW check out SQLite's full text search extension: https://www.sqlite.org/fts5.html

Re: Accidental database programming

#28

Post by obeavs »

bob1029 wrote: post #16

Trying to synchronize state between client & server is a cursed problem. You can sidestep it altogether if you make mild UX sacrifices and revert to something more akin to the PHP/SSR model. SPA is nice, but multipart form posts still work. Just the tiniest amount of javascript can smooth out most of the remaining rough edges. Our latest web products utilize the following client-side state: 3rd party IdP claims for a…

ElectricSQL has made massive strides towards solving this fwiw. Write sqllite in client, guarantee sync to postgres.

See: https://news.ycombinator.com/item?id=37584049

Re: Accidental database programming

#29

Post by padjo »

Many times the thought “what if we just shipped the database to the client” has crossed my mind in large multi tenant apps where individual datasets were relatively small. I’ve never gone far with it as it seems sufficiently outside the norm to be a cursed architectural pattern. Would be nice to find out I was wrong.

Re: Accidental database programming

#30

Post by neilk »

0xbadcafebee wrote: post #20

Anyone remember when "frontend applications" were actual applications and not web pages? I'm willing to bet we have reached that point where new devs literally do not remember that time. There comes a time [..] where we [..] need to cache data from an API. It might start off benign – storing a previous page of data for that instant back button experience, implementing a bit of undo logic, or merging some state from d…

You’re not wrong, but the web won because it had a superior delivery system: URLs. Everything weird about the web era of development has been about contorting everything to be URL-oriented.

But consider how WASM is now turning the browser into an app delivery client. Not a “html and json bodged into an app”, but a real honest to god app.

This project happens to be browser based because that’s convenient place to put a WASM app, and it has a decent presentation layer. But it doesn’t have to be!

Post reply on HN