[deleted]
Sqlite3 WebAssembly
161–170 of 200 posts
Re: Sqlite3 WebAssembly
#162Earlier quoted context omitted.
A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). Adding 400 for such a high quality piece of DB actually borders reasonability. And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.
> what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.
You're comparing 2 Mb of useless and broken animated scrolling with 400 Kb of SQLite, which tells me you have no idea what's behind SQLite's 400 kb.
High quality software is something I rarely see nowadays when browsing "modern" websites.
Re: Sqlite3 WebAssembly
#163Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…
I didn't fully understand this idea of hydration but I also got recently interested in leveraging Wasm, in context of running parts of backend logic using it (eg. regular templating and some Htmx endpoints) to allow certain offline features with regular server-side web frameworks that could use similar Sqlite Wasm datastore as OP.
- Full Django example https://github.com/m-butterfield/django_webassembly - Minimal Rust example with client-server separation using macros https://github.com/richardanaya/wasm-service - Clojure Electric using more macros for something similar (without wasm) https://github.com/hyperfiddle/electric - Recent M$ web tooling starting with letter B?
In last few years, surprisingly many parts of Python ecosystem to have gained Wasm support that can leverage this also. Obvious ML related ones and even game libraries like Pygame and Pyxels. Kivy support (multi-touch UI library) should not be too far out either now that SDL2 library has added support.
Seems like there's many new possibilities for running code in any computer, optionally sandboxed in web browser to avoid platform bureaucracy or taxes.
Re: Sqlite3 WebAssembly
#164Earlier quoted context omitted.
hi simon. i direct messaged you on twitter about a PoC i did of this in aug 2022, but never heard back - i thought you might have been interested. my twitter handle is justjs14. i have some code i would have to dig out that did this very thing - it allows you to open a SQLite db in browser using sqlite (with a VFS) compiled to wasm (not the official WASM build), make changes and both push and pull WALs to and from a…
> (or indeed browser to browser would be possible both manually or over WebRTC) I have just done something similar in the past week, but without the WAL. It's pretty much an alternative to online spreadsheets for me. http://github.com/adhamsalama/sqlite-wasm-webrtc
This is a very interesting demo.
Re: Sqlite3 WebAssembly
#165Earlier quoted context omitted.
Thank you! And side note on your last point - I've been burned too many times by confident hallucinations to trust my foundational learning to GPT. I hope someday that will improve, but for now ChatGPT is as trustworthy as an evening chat with someone at the bar. ... Someone who has been drinking since happy hour.
If you'd like a trustworthy overview, the book Designing Data-Intensive Applications by Martin Kleppmann is a classic. I really hope we get an updated version, but the fundamentals all still hold anyway.
O'Reilly shows a 2nd edition slated for December 2025 but it seems like you can access it early with Safari Books.
https://www.oreilly.com/library/view/designing-data-intensiv...
Re: Sqlite3 WebAssembly
#166[deleted]
Re: Sqlite3 WebAssembly
#167Earlier quoted context omitted.
I… really don't get this. The people on SQLite, employed to work on this full time for over a year, have this to say (on the link I posted above): “Because the WASM build does not have shared memory APIs, activating WAL requires that a client specifically activate exclusive-locking mode for a db handle immediately after opening it, before doing anything else with it… “WAL mode does not provide any concurrency benefit…
> WAL mode does not provide any concurrency benefits in this environment. Except we aren't interested in concurrency or performance benefits of WAL; we want it for something else entirely (replication).
Then you don't need WAL. I didn't say replication isn't possible, I said "that's not how it works."
A bunch of different people solved this already (though none in a couple of weekends); common to all of them: they don't use WAL mode.
Re: Sqlite3 WebAssembly
#168Earlier quoted context omitted.
> WAL mode does not provide any concurrency benefits in this environment. Except we aren't interested in concurrency or performance benefits of WAL; we want it for something else entirely (replication).
Then you don't need WAL. I didn't say replication isn't possible, I said "that's not how it works." A bunch of different people solved this already (though none in a couple of weekends); common to all of them: they don't use WAL mode.
> The shadow WAL is a directory next to your SQLite database where WAL files are effectively recreated as a sequence. . . These WAL files contain the original WAL frames & checksums to ensure consistency.
https://litestream.io/how-it-works/
> (though none in a couple of weekends)
As I said, getting sqlite in wasm to run in WAL mode such that we could start to implement replication, like for example how litestream does. Not this strawman you build, of implementing replication.
Re: Sqlite3 WebAssembly
#169Why have this instead of making it a browser API?
Re: Sqlite3 WebAssembly
#170Earlier quoted context omitted.
> (or indeed browser to browser would be possible both manually or over WebRTC) I have just done something similar in the past week, but without the WAL. It's pretty much an alternative to online spreadsheets for me. http://github.com/adhamsalama/sqlite-wasm-webrtc
cool. will check this out. i think it's an interesting approach and allows all sorts of very low rent interactivity as long as you don't need super high throughput or expect lots of contention.
Thanks!
I purposefully replicate data and queries only, so to change the shown data for everyone you have to run a query, so after the import process (which can be slow for large files), queries are replicated fast enough IMO because it's just a small string sent over WebRTC, so in theory it should be faster to run a query over a big dataset and see its result appear for all peers than a centralized app like Google Sheets, but I haven't benchmarked it yet.