Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

101–110 of 200 posts

Re: Sqlite3 WebAssembly

#101

Post by outlore »

ochiba wrote: post #17
outlore wrote: post #5

i’ve been looking for a Tanstack Query style library that is backed by Sqlite (backed by OPFS or some other browser storage) and syncs with an API in the background. Does anything like that exist? i’ve seen ElectricSQL and other sync engines but they are a bit opinionated. I’m pretty new to local-first but i feel like the developer ergonomics are not quite there yet Meanwhile for “local-only” it would be great to use…

Not sure if you've looked at PowerSync yet: https://www.powersync.com/ (I'm on the team) For the read path it hooks into Postgres logical replication or MongoDB change streams (and MySQL binlog soon). It supports partial syncing using declarative rules. For the write path, it allows writing to the local SQLite database and also places writes into an upload queue, and then uses a developer-defined function to upload w…

ooo i haven’t! will check PowerSync out :)

Re: Sqlite3 WebAssembly

#102

Post by andrewf »

ncruces wrote: post #92
jauntywundrkind wrote: post #83

Earlier quoted context omitted.

Then don't use in-memory sqlite? Use file backed sqlite but have your wasm implementation of those "system calls" just be to memory? I dunno, feels like you're coming down too hard.

File backed SQLite in a browser? Do you mean like OPFS? https://sqlite.org/wasm/doc/trunk/persistence.md#opfs-wal Again, just because the all the Lego pieces sound like they should all just fit together, doesn't mean that they will. The VFS mechanism was primarily designed to make SQLite easy to port to multiple OSes. WAL mode is hard to port everytime you step away from a more traditional OS. “We have SQLite in the…

I haven't looked but I bet a lot of the WAL complexity comes down to supporting consistency and durability guarantees, neither of which you necessarily need for your in-browser use case.

Re: Sqlite3 WebAssembly

#103

Post by chrysoprace »

jamesgpearce wrote: post #93
chrysoprace wrote: post #84

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed anoth…

Disclaimer: I'm the author. But you might be interested in TinyBase.

Thanks I'll give it a look! I guess I really was fishing for someone to make a recommendation and I see you have a lot of backend persistence options which I'm very excited about.

Another issue I have with a lot of the new local first products is that they tend to lock you into a particular database type on the backend, so this is refreshing.

Re: Sqlite3 WebAssembly

#104

Post by insipx »

tonygiorgio wrote: post #38
brandonpollack2 wrote: post #28

I was trying to get this working in a rust ecosystem some time ago but none of the blessed.rs sql (rusqlite, sqlx) wrappers seem to take advantage of it yet and wrapping it yourself is a bit tricky since when I was trying I couldn't figure out a way to to get emscripten wasm code to play nice with wasm32-unknown-unknown without some kind of JS wrapper which then requires implementing the interface those crates expect…

Yeah I’ve been waiting awhile for this myself. A few PRs with work pending for a year or so. I’ve seen some proof of concepts but nothing anywhere close to usable.

you should check out https://github.com/xmtp/diesel-wasm-sqlite

reliable so far, being dogfooded in production as we speak

Re: Sqlite3 WebAssembly

#106

Post by T-Winsnes »

dumbo-octopus wrote: post #97
T-Winsnes wrote: post #88

Earlier quoted context omitted.

The security model is challenging, as it relies on Postgres users for iam. Your users essentially log directly into your db

Isn’t Postgres a fairly capable IAM provider, all things considered? I’d their access control mechanisms at least as much as a run of the mill external backend’s.

For basic auth it works well, but the challenge comes when you need to integrate with oidc, need to enforce mfa, enable sso etc. session invalidation is also quite complicated.

You need an identity middle man in front of the Postgres identity to tackle these and validate that the session is still active. Last time I looked at electric it was a big challenge to integrate such a service. This might have improved since then however

Re: Sqlite3 WebAssembly

#107

Post by Fabricio20 »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

I think OrbitDB [1] is the closest we have to a functional version of this right now? Unsure, but the concept is really cool!

[1]: https://github.com/orbitdb/orbitdb/blob/main/docs/GETTING_ST...

Re: Sqlite3 WebAssembly

#108

Post by billywhizz »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

hi simon. i direct messaged you on twitter about a PoC i did of this in aug 2022, but never heard back - i thought you might have been interested. my twitter handle is justjs14.

i have some code i would have to dig out that did this very thing - it allows you to open a SQLite db in browser using sqlite (with a VFS) compiled to wasm (not the official WASM build), make changes and both push and pull WALs to and from a server (or indeed browser to browser would be possible both manually or over WebRTC). it even works with github pages if you give the browser client a github token to work with.

if you are interested, feel free to ping me and i can see if i can get this up and running from scratch again. i did a ton of experiments with this approach around then and i think it could be useful for a subset of applications at least.

there's also a working demo of the pull functionality only here: https://just.billywhizz.io/sqlite/demo/#https://just.billywh...

Re: Sqlite3 WebAssembly

#109

Post by xyc »

simonw wrote: post #63
parhamn wrote: post #55

WebSQL should've just been Sqlite and the whole offline-first (and general app storage) ecosystem would've been so much nicer. Is there any hope of that happening? Instead of abstracting and over specifying sqlite, can the spec just specify a version of the SQLite API browsers should support and roll the version periodically?

I for one am glad WebSQL didn't establish itself. Now we get the most recent version of SQLite when we need it as a 410KB compressd WASM blob, as opposed to being stuck on browser-mandated versions of SQLite that might even be a decade old at this point.

They are not mutually exclusive though. WebSQL doesn't prevent anyone from loading a WASM blob. And while moving slowly, the browsers does deprecate old stuff and update implementation.

Re: Sqlite3 WebAssembly

#110

Post by koolala »

The CORS restrictions / needing SharedArrayBuffer support kinda stinks.

There is no way to use Sqlite3 off-thread without memory sharing? Couldn't postMessage work to pass data to the sqlite thread by using the third Transfer argument?

Would postMessage transfer allow memory to be stored in a sqlite wasm database running a worker off-thread?

Refering to this implementation's docs: https://github.com/sqlite/sqlite-wasm

Post reply on HN