Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

131–140 of 200 posts

Re: Sqlite3 WebAssembly

#131

Post by nesarkvechnep »

hiccuphippo wrote: post #130
simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

This sounds like what meteor was 10+ years ago but with sqlite instead of mongodb.

Nothing like it, actually.

Re: Sqlite3 WebAssembly

#132

Post by pdyc »

TiredGuy wrote: post #3

So after downloading from the official downloads page and stripping away all the mjs files and "bundler-friendly" files, a minimal sqlite wasm dependency will be about 1.3MB. For an in-browser app, that seems a bit much but of course wasm runs in other places these days where it might make more sense.

That's correct, people in this thread are comparing single compressed dependency of sqlite+wasm of 400KB to the total size of web pages which run in MB. I did some actual tests while trying to use sqlite and it does adds noticeable delay on first page load on mobile due to big size+decompression+ additional scaffolding of wasm. Pages that run into MB have small files that are downloaded concurrently so the delay is not noticeable. I wrote about this and my other expriments with in browser db in my last article but it did not get any traction here.

Re: Sqlite3 WebAssembly

#133

Post by pdyc »

jjcm wrote: post #30

As a general question, in what scenarios is it more beneficial to send the full DB and let the browser handle the queries? Maybe phrased a better way - when would I use this to improve a user experience over the traditional server-hosted db model?

i am creating host of dashboards which directly talk to different services with very little data on my own server that is used for access control and token management only so actual data never comes to my servers. This kind of app is a good candidate for client side embedded db.

Re: Sqlite3 WebAssembly

#134

Post by [deleted] »

chrysoprace wrote: post #84

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed anoth…

[deleted]

Re: Sqlite3 WebAssembly

#135

Post by chrysoprace »

gavmor wrote: post #119
chrysoprace wrote: post #84

I've been really interested in the local-first landscape lately but embedding SQLite seems really heavy-weight compared to using the browser's built-in storage APIs (in particular, IndexedDB) and it seems to be what most of the main open source libraries do. I'm interested to see a open-source solution (with sync) which provides an SQLite-like API but for the browser's native storage rather than trying to embed anoth…

SQL is, arguably, more ergonomic than IndexedDB APIs, and may take up less RAM/CPU when, eg, using a `WHERE` clause rather than an `if() ` filter.

It's definitely more ergonomic, but the browser doesn't have an SQL API where as it does have IndexedDB. I'm hoping for tooling that makes IndexedDB more ergonomic while having a local-first syncing solution.

Dexie gives an ORM-like experience but their syncing solution is not FOSS.

Re: Sqlite3 WebAssembly

#136

Post by asdfman123 »

sgbeal wrote: post #117
catapart wrote: post #62

Earlier quoted context omitted.

Right but, to my eyes, that's vague? What I'm asking is if I need to manage the sqlite file, as I would on an OS's file system, or if accessing the sqlite library will automatically persist that data to those web-native storages, like the way indexedDB doesn't require me to load an "idb" file and then "save" or "commit" that save. I just access it and write. To be clear: I'm not asking academically. I wrote a whole l…

> Right but, to my eyes, that's vague? We (the sqlite project, where the "vague" description comes from) do not define the use cases. Similarly, in the docs for the C library you won't find any more than passing references to specific use cases, and those are typically contrived for the sake of example. (One notable exception: https://sqlite.org/appfileformat.html >) > What I'm asking is if I need to manage the sqlit…

> do not define the use cases

I genuinely don't mean to sound rude, and maybe I misunderstand, but how do you build software if you're not doing it with use cases in mind?

Re: Sqlite3 WebAssembly

#137

Post by globular-toast »

digdugdirk wrote: post #98
wyager wrote: post #86

Earlier quoted context omitted.

WAL: Write ahead log, common strategy for DBs (sqlite, postgres, etc.) to improve commit performance. Instead of fsync()ing every change, you just fsync() a log file that contains all the changes and then you can fsync() the actual changes at your leisure Shared memory API: If you want to share (mutable) data between multiple processes, you need some kind of procedure in place to manage that. How do you get a referen…

Thank you! And side note on your last point - I've been burned too many times by confident hallucinations to trust my foundational learning to GPT. I hope someday that will improve, but for now ChatGPT is as trustworthy as an evening chat with someone at the bar. ... Someone who has been drinking since happy hour.

If you'd like a trustworthy overview, the book Designing Data-Intensive Applications by Martin Kleppmann is a classic. I really hope we get an updated version, but the fundamentals all still hold anyway.

Re: Sqlite3 WebAssembly

#138

Post by gagik_co »

outlore wrote: post #129
gagik_co wrote: post #21

Earlier quoted context omitted.

I have working to replicate TanStack query experience by writing my own queries wrapped around PowerSync, although actually on Flutter (using Flutter Hooks! which was cool to use coming from React). It’s a very internal design meant for my app tetr[1] right now (and actually being migrated over from Realm). I am hoping to potentially standardize it and publish a package once it’s mature enough but not too wrap to mak…

very cool product page!

thank you!

Re: Sqlite3 WebAssembly

#139

Post by matharmin »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

Unlike other opinions here I do think it is technically feasible to stream a copy of the WAL - it just has to be implemented in the VFS. "Shared memory" could be a SharedArrayBuffer, or just a normal buffer if you only have one database connection open at a time (for example in a SharedWorker, which is already common). It may not be simple to implement, but definitely possible.

The biggest actual issue is that it will capture block-level changes, not row-level changes. This means it can work to replicate a complete database, but partial sync (e.g. sharing some rows with other users) won't be feasible.

To get row-level changes for partial sync, you need to use something like triggers or the SQLite session extension [1]. For PowerSync we just use triggers on the client side. I find that works really well, and I haven't found any real downsides to that except perhaps for the work of maintaining the triggers.

[1]: https://sqlite.org/sessionintro.html

Re: Sqlite3 WebAssembly

#140

Post by manmal »

asdfman123 wrote: post #136
sgbeal wrote: post #117

Earlier quoted context omitted.

> Right but, to my eyes, that's vague? We (the sqlite project, where the "vague" description comes from) do not define the use cases. Similarly, in the docs for the C library you won't find any more than passing references to specific use cases, and those are typically contrived for the sake of example. (One notable exception: https://sqlite.org/appfileformat.html >) > What I'm asking is if I need to manage the sqlit…

> do not define the use cases I genuinely don't mean to sound rude, and maybe I misunderstand, but how do you build software if you're not doing it with use cases in mind?

Usage is laid out well in the docs, I‘m not sure GP has read them. IMO it’s obvious that many use cases have been kept in mind. Defining a use case != accommodating a use case
Post reply on HN