Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

111–120 of 200 posts

Re: Sqlite3 WebAssembly

#111

Post by conradev »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

LiteVFS can be compiled in a browser and sync with LiteFS cloud: https://github.com/superfly/litevfs

It needs to be run from a worker, though

Re: Sqlite3 WebAssembly

#112

Post by sgbeal »

samatman wrote: post #90
sgbeal wrote: post #72

Earlier quoted context omitted.

> A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework It's an out-of-language packaging/distribution framework (and it's not the only one). It's not part of the JS standards. My comments above have been edited to reframe our stance on npm and frameworks in general.

I think the communication barrier here is that in JavaScript, framework very distinctly means things like React, Vue, Angular, and so on. It definitely does not refer to projects like Node/npm/Bun/Deno, those are toolchains, sometimes called ecosystems for obscure reasons. If you changed the word "framework" to "toolchain" in your post I think it would make a lot more sense to people.

> If you changed the word "framework" to "toolchain" in your post I think it would make a lot more sense to people.

Fair point but the edit window has passed ;). For the sake of clarity for those still following along: "framework," in the context of my above comments, includes any non-formally-standardized tools or APIs which are built atop the standardized core.

Re: Sqlite3 WebAssembly

#113

Post by azakai »

brandonpollack2 wrote: post #28

I was trying to get this working in a rust ecosystem some time ago but none of the blessed.rs sql (rusqlite, sqlx) wrappers seem to take advantage of it yet and wrapping it yourself is a bit tricky since when I was trying I couldn't figure out a way to to get emscripten wasm code to play nice with wasm32-unknown-unknown without some kind of JS wrapper which then requires implementing the interface those crates expect…

> I couldn't figure out a way to to get emscripten wasm code to play nice with wasm32-unknown-unknown

There is good news there, some people plan to get Emscripten and Rust to work well together in Wasm:

https://github.com/rustwasm/wasm-bindgen/pull/4014#issuecomm...

Re: Sqlite3 WebAssembly

#114

Post by matlin »

outlore wrote: post #5

i’ve been looking for a Tanstack Query style library that is backed by Sqlite (backed by OPFS or some other browser storage) and syncs with an API in the background. Does anything like that exist? i’ve seen ElectricSQL and other sync engines but they are a bit opinionated. I’m pretty new to local-first but i feel like the developer ergonomics are not quite there yet Meanwhile for “local-only” it would be great to use…

I've made just this!

Docs for it: https://www.triplit.dev/docs/frameworks/tanstack-router#exam...

It by default uses IndexedDB but can also use SQLite but does real time, relational querying and (optionally) syncs with you server.

Re: developer ergonomics, this is our primary focus so I don't love to get your feedback on!

Re: Sqlite3 WebAssembly

#115

Post by sgbeal »

Ciantic wrote: post #16
simonw wrote: post #4

Slight point of confusion: that page says: > These components were initially released for public beta with version 3.40 and will tentatively be made API-stable with the 3.41 release, pending community feedback. But the most recent release of SQLite is 3.46.1 (from 2024-08-13) Presumably they are now "API-stable" but the page hasn't been updated yet. It would be great if the SQLite team published an official npm packa…

> It would be great if the SQLite team published an official npm package bundling the WASM version, could be a neat distribution mechanism for them. I think they've been doing that for a while, in JS script you can already do this: import sqlite3InitModule from "https://cdn.jsdelivr.net/npm/@sqlite.org/sqlite-wasm/sqlite-wasm/jswasm/sqlite3-bundler-friendly.mjs"; const sqlite3 = await sqlite3InitModule({ locateFile(f…

> > It would be great if the SQLite team published an official npm package

> I think they've been doing that for a while,

Kinda: https://sqlite.org/wasm/doc/trunk/npm.md>

We in the sqlite project neither use nor require npm in any capacity whatsoever, so it would be kinda silly for us to attempt to support it. We instead leave that level of code/tools to folks who use and/or care about them.

There _is_ an "officially sanctioned/blessed" npm repo, and we actively support its maintainer (e.g. we participate the issue tracker and make patches in the core distribution where they're strictly needed), but we otherwise keep a "hands off" policy when it comes to non-standardized APIs and toolchains.

We _like_ to see people to plug the sources into their tools of choice, but we cannot feasibly take on the burden of doing that plugging-in for them, especially given how fluid the JavaScript ecosystem is when it comes to frameworks and tools.

Sidebar: we rely heavily on Emscripten because there is, for all practical purposes, it has no substitute, but we also actively go out of our way to ensure that the sources can be easily plugged in to an alternative should one ever appear.

Re: Sqlite3 WebAssembly

#116

Post by sgbeal »

koolala wrote: post #110

The CORS restrictions / needing SharedArrayBuffer support kinda stinks. There is no way to use Sqlite3 off-thread without memory sharing? Couldn't postMessage work to pass data to the sqlite thread by using the third Transfer argument? Would postMessage transfer allow memory to be stored in a sqlite wasm database running a worker off-thread? Refering to this implementation's docs: https://github.com/sqlite/sqlite-was…

> The CORS restrictions / needing SharedArrayBuffer support kinda stinks.

We have no CORS restrictions but one specific (and optional) VFS requires COOP/COEP for SharedArrayBuffer. If SharedArrayBuffer isn't available, that VFS won't load, but the rest of the library will plod along just fine:

https://sqlite.org/wasm/doc/tip/persistence.md

Re: Sqlite3 WebAssembly

#117

Post by sgbeal »

catapart wrote: post #62
azangru wrote: post #57

Earlier quoted context omitted.

From the about page: > Specific Goals of this Project > Insofar as possible, support persistent client-side storage using available JS APIs. As of this writing, that includes the Origin-Private FileSystem (OPFS) and (very limited) storage via the window.localStorage and window.sessionStorage backend.

Right but, to my eyes, that's vague? What I'm asking is if I need to manage the sqlite file, as I would on an OS's file system, or if accessing the sqlite library will automatically persist that data to those web-native storages, like the way indexedDB doesn't require me to load an "idb" file and then "save" or "commit" that save. I just access it and write. To be clear: I'm not asking academically. I wrote a whole l…

> Right but, to my eyes, that's vague?

We (the sqlite project, where the "vague" description comes from) do not define the use cases. Similarly, in the docs for the C library you won't find any more than passing references to specific use cases, and those are typically contrived for the sake of example. (One notable exception: https://sqlite.org/appfileformat.html>)

> What I'm asking is if I need to manage the sqlite file, as I would on an OS's file system, or if accessing the sqlite library will automatically persist that data to those web-native storages, like the way indexedDB doesn't require me to load an "idb" file and then "save" or "commit" that save. I just access it and write.

That's all covered in the docs (of which there are well more than 100 lovingly-hand-written pages), but the short answer is "it just works." You have the _option_ of importing and exporting databases from the browser-native storage, but you don't have to.

For starters, see: https://sqlite.org/wasm/doc/tip/persistence.md>

Re: Sqlite3 WebAssembly

#118

Post by sgbeal »

deskr wrote: post #95
TiredGuy wrote: post #3

So after downloading from the official downloads page and stripping away all the mjs files and "bundler-friendly" files, a minimal sqlite wasm dependency will be about 1.3MB. For an in-browser app, that seems a bit much but of course wasm runs in other places these days where it might make more sense.

Sadly, 1.3 MB is nothing on the modern web, especially for a static file. BBC's frontpage loads 3.78 MB. https://www.bbc.co.uk/

> BBC's frontpage loads 3.78 MB.

FWIW: Google Drive just downloaded 15.4mb to boot up for me and imdb dot com hit some 7+mb before it started auto-loading videos on top of that.

Re: Sqlite3 WebAssembly

#119

Post by gavmor »

chrysoprace wrote: post #84

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed anoth…

SQL is, arguably, more ergonomic than IndexedDB APIs, and may take up less RAM/CPU when, eg, using a `WHERE` clause rather than an `if() ` filter.

Re: Sqlite3 WebAssembly

#120

Post by sgbeal »

hawski wrote: post #46
TiredGuy wrote: post #3

So after downloading from the official downloads page and stripping away all the mjs files and "bundler-friendly" files, a minimal sqlite wasm dependency will be about 1.3MB. For an in-browser app, that seems a bit much but of course wasm runs in other places these days where it might make more sense.

Is there a way to statically compile an application with SQLite and the result WASM was smaller. So for example I have an app that would use only a specific subset of SQLite. Could the SQLite's WASM be built with this in mind cutting down on code that is not used? Or is there a way to prune it having the used API surface? In a regular compiler/linker scenario it would just be a static compilation. Here we have a JS a…

> Could the SQLite's WASM be built with this in mind cutting down on code that is not used?

The pending 3.47 release has some build-side tweaks which enable a user to strip it down to "just the basics," but we've not yet been able to get it smaller than about 25-30% less than it otherwise is:

    cd ext/wasm
    make barebones=1 ; # requires GNU Make and the Emscripten SDK
Doing that requires building it yourself - there are no plans to publish deliverables built that way.

The build process also supports including one's own C code, which could hypothetically be used to embed an application and the wasm part of the library (as distinct from the JS part) into a single wasm file. Its primary intended usage is to add SQLite extensions which are not part of the standard amalgamation build.

> Or is there a way to prune it having the used API surface?

Not with the provided JS pieces. Those have to expose essentially the whole C library, so they will not be pruned from the wasm file.

However, you could provide your own JS bindings which only use a small subset of the API, and Emscripten is supposedly pretty good about stripping out C-side code which neither explicitly exported nor referenced anywhere. You'd be on your own - that's not something we'll integrate into the canonical build process - but we could provide high-level support, via the project's forum, for folks taking that route.

Post reply on HN