Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

91–100 of 200 posts

Re: Sqlite3 WebAssembly

#91

Post by wahern »

rmbyrro wrote: post #60
jsheard wrote: post #19

Earlier quoted context omitted.

It's pretty compressible at least, sqlite3.js+wasm are 1.3MB raw but minifying the JS and then compressing both files with Brotli gets them down to 410KB.

A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). Adding 400 for such a high quality piece of DB actually borders reasonability. And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.

> what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable

They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.

Re: Sqlite3 WebAssembly

#92

Post by ncruces »

jauntywundrkind wrote: post #83
ncruces wrote: post #68

Earlier quoted context omitted.

There are many layers of that's not how it works at play here. In-memory SQLite databases don't use WAL. Wasm (and browser Wasm, in particular) doesn't support anything like the shared memory APIs SQLite wants for its WAL mode. Litestream requires a very precise WAL setup to work (which just so happens to work with the default native SQLite setup, but is hard to replicate with Wasm). Cloudflare Durable Objects may ha…

Then don't use in-memory sqlite? Use file backed sqlite but have your wasm implementation of those "system calls" just be to memory? I dunno, feels like you're coming down too hard.

File backed SQLite in a browser? Do you mean like OPFS?

https://sqlite.org/wasm/doc/trunk/persistence.md#opfs-wal

Again, just because the all the Lego pieces sound like they should all just fit together, doesn't mean that they will.

The VFS mechanism was primarily designed to make SQLite easy to port to multiple OSes. WAL mode is hard to port everytime you step away from a more traditional OS.

“We have SQLite in the browser, let's just stream the WAL like the Litestream hack” does not add up.

It's not impossible, but it surely took a lot of effort at Cloudflare (or at Fly, or Turso) to get there. And neither of them opened it up sufficiently to help us do the same. In a browser.

Re: Sqlite3 WebAssembly

#93

Post by jamesgpearce »

chrysoprace wrote: post #84

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed anoth…

Disclaimer: I'm the author. But you might be interested in TinyBase.

Re: Sqlite3 WebAssembly

#94

Post by stackskipton »

6gvONxR4sf7o wrote: post #74
stackskipton wrote: post #40

Earlier quoted context omitted.

SRE here, it's currently happening in a few parts but overall, it's not as attractive on server side. Server Side code running is mostly a solved problem and for very few organizations, the benefits of WASM don't outweigh any difficulties in getting it running.

> Server Side code running is mostly a solved problem I know what you mean here, but I think we're very limited in what we tend to run. Polyglot programming still isn't really a thing, and with things like WASI standardized (someday soon I hope), I could imagine it becoming a lot nicer.

I feel like Polyglot programming in single app sounds like a nightmare but as non dev, meh, whatever.

Re: Sqlite3 WebAssembly

#95

Post by deskr »

TiredGuy wrote: post #3

So after downloading from the official downloads page and stripping away all the mjs files and "bundler-friendly" files, a minimal sqlite wasm dependency will be about 1.3MB. For an in-browser app, that seems a bit much but of course wasm runs in other places these days where it might make more sense.

Sadly, 1.3 MB is nothing on the modern web, especially for a static file. BBC's frontpage loads 3.78 MB.

https://www.bbc.co.uk/

Re: Sqlite3 WebAssembly

#96

Post by jsheard »

rmbyrro wrote: post #60
jsheard wrote: post #19

Earlier quoted context omitted.

It's pretty compressible at least, sqlite3.js+wasm are 1.3MB raw but minifying the JS and then compressing both files with Brotli gets them down to 410KB.

A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). Adding 400 for such a high quality piece of DB actually borders reasonability. And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.

> A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!).

I think you can probably blame Tailwind for that.

Re: Sqlite3 WebAssembly

#97

Post by dumbo-octopus »

T-Winsnes wrote: post #88
PUSH_AX wrote: post #76

Earlier quoted context omitted.

What’s the catch with this thing?

The security model is challenging, as it relies on Postgres users for iam. Your users essentially log directly into your db

Isn’t Postgres a fairly capable IAM provider, all things considered? I’d their access control mechanisms at least as much as a run of the mill external backend’s.

Re: Sqlite3 WebAssembly

#98

Post by digdugdirk »

wyager wrote: post #86
digdugdirk wrote: post #82

Earlier quoted context omitted.

As someone who uses sqlite fairly regularly, but doesn't understand what most of those paragraphs mean, do you have any recommendations for learning resources? I'm gathering that I need to learn about: - WAL - Shared Memory APIs - Concurrency models - Durable Objects?

WAL: Write ahead log, common strategy for DBs (sqlite, postgres, etc.) to improve commit performance. Instead of fsync()ing every change, you just fsync() a log file that contains all the changes and then you can fsync() the actual changes at your leisure Shared memory API: If you want to share (mutable) data between multiple processes, you need some kind of procedure in place to manage that. How do you get a referen…

Thank you!

And side note on your last point - I've been burned too many times by confident hallucinations to trust my foundational learning to GPT. I hope someday that will improve, but for now ChatGPT is as trustworthy as an evening chat with someone at the bar.

... Someone who has been drinking since happy hour.

Re: Sqlite3 WebAssembly

#99

Post by runarberg »

I’m working on a hobby-project that uses IndexedDB for persistent client-side storage, and it really feels like W3C made some very bad design decision and than instead of fixing they they have just given up on the standard. Issues like not being able to index values in objects in arrays [1] (not even in fixed position e.g. "key.path.[0].value") despite almost a decade of developers asking for it, a very limited query syntax, and even the documentation on MDN seems of very lower quality than the rest of the web docs.

I’m happy that we are actually be able to use SQL in the browser now (although I would rather skip the MBs of the bundle bloat). But I feel like the standards committee will now have even less of a reason to fix the very broken state of IndexedDB.

1: https://github.com/w3c/IndexedDB/issues/35

Re: Sqlite3 WebAssembly

#100

Post by syndicatedjelly »

rezonant wrote: post #85
sgbeal wrote: post #72

Earlier quoted context omitted.

> A bit confused at this, NPM is just a package manager / distribution mechanism, not a framework It's an out-of-language packaging/distribution framework (and it's not the only one). It's not part of the JS standards. My comments above have been edited to reframe our stance on npm and frameworks in general.

I don't think there will ever be a package manager dictated by the Ecmascript standards.

Then ES will continue to remain an outlier among major language implementations
Post reply on HN