Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

81–90 of 200 posts

Re: Sqlite3 WebAssembly

#81

Post by akira2501 »

debugnik wrote: post #58
joemi wrote: post #51

Earlier quoted context omitted.

I wonder why it was unmaintained/dropped. Was there something wrong with it, and if so, would that also apply to this kind of wasm implementation?

Mozilla refused to support it because then every implementation would have simply used SQLite, which would have promoted any implementation details to a de facto standard. (Even caniuse erroneously describes the feature as "allows SQLite database queries".) From the latest spec [1]: > The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple indepe…

Did they really assume that they were going to be able to _restandardize_ SQL? No wonder IndexDB is hot useless garbage.

The standardization issues around SQL already exist, are already widely known, and where common workarounds are already in practice. It's also an open source project that could have _easily_ incorporated compatibility code for this specific use case anyways.

They made blind fealty to process more important than the outcome to end users.

What a waste.

Re: Sqlite3 WebAssembly

#82

Post by digdugdirk »

ncruces wrote: post #68
simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

There are many layers of that's not how it works at play here. In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode. Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm). Cloudflare Durable Objects may ha…

As someone who uses sqlite fairly regularly, but doesn't understand what most of those paragraphs mean, do you have any recommendations for learning resources?

I'm gathering that I need to learn about: - WAL - Shared Memory APIs - Concurrency models - Durable Objects?

Re: Sqlite3 WebAssembly

#83

Post by jauntywundrkind »

ncruces wrote: post #68
simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

There are many layers of that's not how it works at play here. In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode. Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm). Cloudflare Durable Objects may ha…

Then don't use in-memory sqlite? Use file backed sqlite but have your wasm implementation of those "system calls" just be to memory?

I dunno, feels like you're coming down too hard.

Re: Sqlite3 WebAssembly

#84

Post by chrysoprace »

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed another executable in Web Assembly.

Re: Sqlite3 WebAssembly

#85

Post by rezonant »

sgbeal wrote: post #72
rezonant wrote: post #69

Earlier quoted context omitted.

> We publish ONLY vanilla JS and adamantly refuse to go down rabit hole of supporting the frameworks du jour A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework. Totally fair if you don't want to publish for all the package managers, though for Javascript there's only a few that are relevant. NPM has been around for a decade.

> A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework It's an out-of-language packaging/distribution framework (and it's not the only one). It's not part of the JS standards. My comments above have been edited to reframe our stance on npm and frameworks in general.

I don't think there will ever be a package manager dictated by the Ecmascript standards.

Re: Sqlite3 WebAssembly

#86

Post by wyager »

digdugdirk wrote: post #82
ncruces wrote: post #68

Earlier quoted context omitted.

There are many layers of that's not how it works at play here. In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode. Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm). Cloudflare Durable Objects may ha…

As someone who uses sqlite fairly regularly, but doesn't understand what most of those paragraphs mean, do you have any recommendations for learning resources? I'm gathering that I need to learn about: - WAL - Shared Memory APIs - Concurrency models - Durable Objects?

WAL: Write ahead log, common strategy for DBs (sqlite, postgres, etc.) to improve commit performance. Instead of fsync()ing every change, you just fsync() a log file that contains all the changes and then you can fsync() the actual changes at your leisure

Shared memory API: If you want to share (mutable) data between multiple processes, you need some kind of procedure in place to manage that. How do you get a reference to the data to multiple processes, how do you make sure they don't trample each other's writes, etc.

Concurrency model: There are many different ways you can formalize concurrent processes and the way they interact (message passing, locking, memory ordering semantics, etc.). Different platforms will expose different concurrency primitives that may not work the same way as other platforms and may require different reasoning or code structure

Durable objects - I think this is some Cloudflare service where they host data that can be read or modified by your users

This is all from memory, but IME, GPT is pretty good for asking about concepts at this level of abstraction

Re: Sqlite3 WebAssembly

#87

Post by jchanimal »

ncruces wrote: post #68
simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

There are many layers of that's not how it works at play here. In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode. Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm). Cloudflare Durable Objects may ha…

If you like, solving these sort of problems, we are tackling them at Fireproof.

Our database API is modeled on CouchDB and MongoDB, but our storage abstractions are along the lines of what you need to build the multi writer WAL you describe.

More details here https://jsr.io/@fireproof/encrypted-blockstore

Re: Sqlite3 WebAssembly

#88

Post by T-Winsnes »

PUSH_AX wrote: post #76
dustinchilson wrote: post #36

Earlier quoted context omitted.

Are you thinking something like https://electric-sql.com/

What’s the catch with this thing?

The security model is challenging, as it relies on Postgres users for iam. Your users essentially log directly into your db

Re: Sqlite3 WebAssembly

#89

Post by koeng »

ncruces wrote: post #80
TN1ck wrote: post #77

Earlier quoted context omitted.

Very cool project! Do you know if this would be possible for duckdb? Is there something about sqlites APIs and wasm build that made it feasible? Context: Currently using go-duckdb and while it's working for us, getting rid of cgo would be a huge help. Would be quite interested myself to attempt this.

I don't know much about DuckDB's architecture. Wasm is fine for compute (though concurrency is still a somewhat open question). To have Wasm talk to the outside world, you need “host calls” where the guest calls the host. On a browser that's Wasm calling JavaScript. On my Go driver, it's Wasm calling Go. For server side, there's also a standard set of “host calls” modeled around POSIX/Linux syscalls called WASI. I co…

> So SQLite in Wasm is just doing compute, and I do all the OS level stuff in Go. No need for Wasm concurrency, cause I can load multiple instances of my Wasm which act like independent OS processes that communicate through the filesystem (SQLite excels at this).

Interesting. So when I am running concurrent readers using your package, it is just loading multiple instances of the wasm code? (I bottleneck to a single writer in the application)

Re: Sqlite3 WebAssembly

#90

Post by samatman »

sgbeal wrote: post #72
rezonant wrote: post #69

Earlier quoted context omitted.

> We publish ONLY vanilla JS and adamantly refuse to go down rabit hole of supporting the frameworks du jour A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework. Totally fair if you don't want to publish for all the package managers, though for Javascript there's only a few that are relevant. NPM has been around for a decade.

> A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework It's an out-of-language packaging/distribution framework (and it's not the only one). It's not part of the JS standards. My comments above have been edited to reframe our stance on npm and frameworks in general.

I think the communication barrier here is that in JavaScript, framework very distinctly means things like React, Vue, Angular, and so on. It definitely does not refer to projects like Node/npm/Bun/Deno, those are toolchains, sometimes called ecosystems for obscure reasons.

If you changed the word "framework" to "toolchain" in your post I think it would make a lot more sense to people.

Post reply on HN