Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

61–70 of 200 posts

Re: Sqlite3 WebAssembly

#61

Post by emn13 »

parhamn wrote: post #55

WebSQL should've just been Sqlite and the whole offline-first (and general app storage) ecosystem would've been so much nicer. Is there any hope of that happening? Instead of abstracting and over specifying sqlite, can the spec just specify a version of the SQLite API browsers should support and roll the version periodically?

"Rolling the version periodically" is probably quite problematic for browsers. Kind of a key point of the web is that stuff if at all possible keeps working. Breaking changes like that are hard.

Even if the spec just listed occasional version and the webpage could choose which one; that means a potentially tricky maintenance burden on browser to support old versions of a potentially no longer supported sqlite, and each version is another megabyte.

Why not then just choose this solution, and let each website pick its own poison?

If the concern is the repeated downloads of common resources, well, we've accepted that for other CDN's too, and a solution for shared caching of common dependencies would in any case be more valuable than merely for sqlite.

The current approach seems better than a browser-provided version.

Re: Sqlite3 WebAssembly

#62

Post by catapart »

azangru wrote: post #57
catapart wrote: post #48

I wasn't able to tell from a quick look through the page: could someone help me understand the use cases here? More specifically, would this be able to be a "replacement" for indexedDB? Does the data persist, or do I need to keep the sqlite file in the filesytemAPI (or indexedDB/localstorage) myself?

From the about page: > Specific Goals of this Project > Insofar as possible, support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) and (very limited) storage via the window.localStorage and window.sessionStorage backend.

Right but, to my eyes, that's vague?

What I'm asking is if I need to manage the sqlite file, as I would on an OS's file system, or if accessing the sqlite library will automatically persist that data to those web-native storages, like the way indexedDB doesn't require me to load an "idb" file and then "save" or "commit" that save. I just access it and write.

To be clear: I'm not asking academically. I wrote a whole library for managing data in indexedDB for local-first apps, and while it works well enough for what I need, it's iDB so it's subject to data deletion (not common, but allowed in the spec if necessary), and it's a pain to work with just because of its nature and API. So I've been waiting to move to sqlite for a while with the only holdbacks being "is it too heavy?", and "how much has to change?". With WASM, I think we're about as lightweight as its going to get. So I'm just curious if this aims to be a drop-in replacement, or if it still expects you to use it like sqlite on a native platform.

Re: Sqlite3 WebAssembly

#63

Post by simonw »

parhamn wrote: post #55

WebSQL should've just been Sqlite and the whole offline-first (and general app storage) ecosystem would've been so much nicer. Is there any hope of that happening? Instead of abstracting and over specifying sqlite, can the spec just specify a version of the SQLite API browsers should support and roll the version periodically?

I for one am glad WebSQL didn't establish itself.

Now we get the most recent version of SQLite when we need it as a 410KB compressd WASM blob, as opposed to being stuck on browser-mandated versions of SQLite that might even be a decade old at this point.

Re: Sqlite3 WebAssembly

#65

Post by aabhay »

brandonpollack2 wrote: post #28

I was trying to get this working in a rust ecosystem some time ago but none of the blessed.rs sql (rusqlite, sqlx) wrappers seem to take advantage of it yet and wrapping it yourself is a bit tricky since when I was trying I couldn't figure out a way to to get emscripten wasm code to play nice with wasm32-unknown-unknown without some kind of JS wrapper which then requires implementing the interface those crates expect…

I have been working on one. If you're interested in working on it or contributing, feel free to chime in here:

https://github.com/rhashimoto/wa-sqlite/discussions/154

This essentially requires that we import the sqlite emscripten build via an extern C header in wasm bindgen, and then we need to re-implement the VFS in rust while compiling it in multi-threaded mode to allow for shared array buffer access. After that is all done, we will be able to access SQLite rows as raw wasm bytes. That gives us the ability to implement a rust-sqlite style wrapper or integration. There would still not be some of the niceties such as connection pooling, but in wasm you likely want to use the db in exclusive mode.

Re: Sqlite3 WebAssembly

#66

Post by xyc »

debugnik wrote: post #58
joemi wrote: post #51

Earlier quoted context omitted.

I wonder why it was unmaintained/dropped. Was there something wrong with it, and if so, would that also apply to this kind of wasm implementation?

Mozilla refused to support it because then every implementation would have simply used SQLite, which would have promoted any implementation details to a de facto standard. (Even caniuse erroneously describes the feature as "allows SQLite database queries".) From the latest spec [1]: > The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple indepe…

i have a feeling that it set back the web by a decade

https://x.com/chxy/status/1822858746307170640

Re: Sqlite3 WebAssembly

#67

Post by sgbeal »

simonw wrote: post #4

Slight point of confusion: that page says: > These components were initially released for public beta with version 3.40 and will tentatively be made API-stable with the 3.41 release, pending community feedback. But the most recent release of SQLite is 3.46.1 (from 2024-08-13) Presumably they are now "API-stable" but the page hasn't been updated yet. It would be great if the SQLite team published an official npm packa…

> Presumably they are now "API-stable" but the page hasn't been updated yet.

That's correct. i'll try my best to remember to update that reference the next time i'm back on the computer.

> It would be great if the SQLite team published an official npm package

Not a chance. We publish only vanilla JS and adamantly refuse to go down the rabit hole of supporting out-of-language tools (none of which any of our project members use). We support an "officially sanctioned" npm build, maintained by Thomas Steiner, but do not actively develop for any JS frameworks.

Direct support for any given framework (npm included) would give the impression that we endorse that framework, and endorsement of third-party projects is something we actively avoid.

Re: Sqlite3 WebAssembly

#68

Post by ncruces »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

There are many layers of that's not how it works at play here.

In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode.

Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm).

Cloudflare Durable Objects may have been inspired by Litestream but works very differently (as do LiteFS, Turso, etc…)

The general idea of streaming changes from SQLite would work, but it's a lot of work, and the concurrency model of in-browser Wasm will make it challenging to implement.

(I wrote that forum post some time ago, and have WAL working in a server side Wasm build of SQLite, but none of the options to make it work would make much sense, or be possible, in browser)

Re: Sqlite3 WebAssembly

#69

Post by rezonant »

sgbeal wrote: post #67
simonw wrote: post #4

Slight point of confusion: that page says: > These components were initially released for public beta with version 3.40 and will tentatively be made API-stable with the 3.41 release, pending community feedback. But the most recent release of SQLite is 3.46.1 (from 2024-08-13) Presumably they are now "API-stable" but the page hasn't been updated yet. It would be great if the SQLite team published an official npm packa…

> Presumably they are now "API-stable" but the page hasn't been updated yet. That's correct. i'll try my best to remember to update that reference the next time i'm back on the computer. > It would be great if the SQLite team published an official npm package Not a chance. We publish only vanilla JS and adamantly refuse to go down the rabit hole of supporting out-of-language tools (none of which any of our project me…

> We publish ONLY vanilla JS and adamantly refuse to go down rabit hole of supporting the frameworks du jour

A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework. Totally fair if you don't want to publish for all the package managers, though for Javascript there's only a few that are relevant. NPM has been around for a decade.

Re: Sqlite3 WebAssembly

#70

Post by insipx »

brandonpollack2 wrote: post #28

I was trying to get this working in a rust ecosystem some time ago but none of the blessed.rs sql (rusqlite, sqlx) wrappers seem to take advantage of it yet and wrapping it yourself is a bit tricky since when I was trying I couldn't figure out a way to to get emscripten wasm code to play nice with wasm32-unknown-unknown without some kind of JS wrapper which then requires implementing the interface those crates expect…

Got this SQLite build working with rusts diesel sqlite here: https://github.com/xmtp/diesel-wasm-sqlite

I'm gearing up for a 0.2 release this week which should iron out a few kinks, but otherwise you can use the diesel ORM as if its native

Post reply on HN