Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sqlite3 WebAssembly

sqlite.org

Post reply on HN
Print view View original

141–150 of 200 posts

Re: Sqlite3 WebAssembly

#141

Post by ncruces »

andrewf wrote: post #102
ncruces wrote: post #92

Earlier quoted context omitted.

File backed SQLite in a browser? Do you mean like OPFS? https://sqlite.org/wasm/doc/trunk/persistence.md#opfs-wal Again, just because the all the Lego pieces sound like they should all just fit together, doesn't mean that they will. The VFS mechanism was primarily designed to make SQLite easy to port to multiple OSes. WAL mode is hard to port everytime you step away from a more traditional OS. “We have SQLite in the…

I haven't looked but I bet a lot of the WAL complexity comes down to supporting consistency and durability guarantees, neither of which you necessarily need for your in-browser use case.

Not really, or it depends. That complexity is dealt with by SQLite.

The complexity for you comes from trying to reuse their (battle tested) implementation when (as I wrote above) the primitives they depend upon were not meant to make porting to a browser sandbox easy.

And the problems there are the specific concurrency model they depend upon: communicate by sharing memory.

Then, you're either working at the wrong abstraction level (and it shows), or you're patching and gutting SQLite.

SQLite is meant to work with files, file locks, shared memory, fsync and mmap.

It also doesn't work out great if you try to persist to an object store, to a KV store, or…

I'm repeating myself, but yeah. You can make it work. Others have made it work. But it's still a lot of work, and you're throwing away a lot of what makes SQLite… SQLite.

Re: Sqlite3 WebAssembly

#142

Post by ncruces »

infogulch wrote: post #125
ncruces wrote: post #73

Earlier quoted context omitted.

Author here. If you're interested, do ask questions.

Hey how's the mmap-based shared memory WAL approach working out? It's been about half a year since you finished the implementation and I only see one issue about it on the tracker, a good sign?

It's turning out great, I guess.

The current approach is not portable to Windows, but it works fine on Linux, macOS, BSD and illumos. In general, portability is hindered more by file locking (I hate POSIX locks) than mmap.

The currently open GitHub issue is more bad default configuration than anything else. Configuring connections to use less memory by default should fix it.

I already have a PR ready for the next release that also opens this up for 32-bit platforms.

Also, GoToSocial (a self-hostable Mastodon alternative) moved to it (from modernc) for its first beta release.

https://github.com/superseriousbusiness/gotosocial

Re: Sqlite3 WebAssembly

#143

Post by Dylan16807 »

wahern wrote: post #91
rmbyrro wrote: post #60

Earlier quoted context omitted.

A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). Adding 400 for such a high quality piece of DB actually borders reasonability. And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.

> what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.

> They're thinking, "adding [some fraction of existing total payload] for such a high quality [feature] actually borders reasonability". Wash. Rinse. Repeat.

Context makes all the difference here. If you're considering a big chunk of size for a relational database engine, you need to ask: are you making a complex application, or a normal web page? If it's the latter, then it's not reasonable at all.

And anything that makes the HTML itself that big is almost certainly bloat, not "high quality", and shouldn't be used in any context.

Re: Sqlite3 WebAssembly

#144

Post by Dylan16807 »

telotortium wrote: post #31
coder543 wrote: post #22

Earlier quoted context omitted.

1.3MB seems perfectly reasonable in a modern web app, especially since it will be cached after the first visit to the site. If you’re just storing user preferences, obviously don’t download SQLite for your web app just to do that… but if you’re doing something that benefits from a full database, don’t fret so much about 1MB that you go try to reinvent the wheel for no reason. If the other comment is correct, then it…

A megabyte here, a megabyte there, pretty soon you’re talking about a really heavyweight app.

By the time you have a good reason to add this library, I think you're already in heavyweight app territory.

Re: Sqlite3 WebAssembly

#146

Post by yoavm »

jsheard wrote: post #96
rmbyrro wrote: post #60

Earlier quoted context omitted.

A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). Adding 400 for such a high quality piece of DB actually borders reasonability. And makes me think: what the hell are frontend devs thinking!? Multiple MB's in JS for a news website. Hundreds of KB's for HTML. It's totally unreasonable.

> A lot of HTML's nowadays have 100 - 300 kb. That's only the HTML (!!). I think you can probably blame Tailwind for that.

Why? More often than not the classes are combined during post-processing to the most reusable unified classes, with very short classes names.

Re: Sqlite3 WebAssembly

#147

Post by ncruces »

koeng wrote: post #89
ncruces wrote: post #80

Earlier quoted context omitted.

I don't know much about DuckDB's architecture. Wasm is fine for compute (though concurrency is still a somewhat open question). To have Wasm talk to the outside world, you need “host calls” where the guest calls the host. On a browser that's Wasm calling JavaScript. On my Go driver, it's Wasm calling Go. For server side, there's also a standard set of “host calls” modeled around POSIX/Linux syscalls called WASI. I co…

> So SQLite in Wasm is just doing compute, and I do all the OS level stuff in Go. No need for Wasm concurrency, cause I can load multiple instances of my Wasm which act like independent OS processes that communicate through the filesystem (SQLite excels at this). Interesting. So when I am running concurrent readers using your package, it is just loading multiple instances of the wasm code? (I bottleneck to a single w…

Yes.

Each connection lives in its own isolated sandbox, and only communicates with other connections through the “file system” (which is a virtual abstraction, actually).

WAL mode is the “exception”: a few pages of the sandbox's memory are mapped to a file, and shared by all connections to the same database.

Each sandbox is single threaded, and mostly lock free, does all its business in the calling goroutine, and regularly checks back with the Go runtime to play nice with the Go scheduler.

It's a bit like an OS running multiple processes, with the VFS layer handling all syscalls.

Re: Sqlite3 WebAssembly

#148

Post by adhamsalama »

billywhizz wrote: post #108
simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

hi simon. i direct messaged you on twitter about a PoC i did of this in aug 2022, but never heard back - i thought you might have been interested. my twitter handle is justjs14. i have some code i would have to dig out that did this very thing - it allows you to open a SQLite db in browser using sqlite (with a VFS) compiled to wasm (not the official WASM build), make changes and both push and pull WALs to and from a…

> (or indeed browser to browser would be possible both manually or over WebRTC)

I have just done something similar in the past week, but without the WAL.

It's pretty much an alternative to online spreadsheets for me.

http://github.com/adhamsalama/sqlite-wasm-webrtc

Re: Sqlite3 WebAssembly

#149

Post by adhamsalama »

simonw wrote: post #32

Something that would be really fun would be to run SQLite in-memory in a browser but use the same tricks as Litestream and Cloudflare Durable Objects ( https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... ) to stream a copy of the WAL log to a server (maybe over a WebSocket, though intermittent fetch() POST would work too). Then on subsequent visits use that server-side data to rehydrate the client-side da…

I used SQLite compiled to WebAssembly in the browser and replicated it (without the WAL) using WebRTC to create a way to collaborate on databases in the browser instead of using apps like Google Sheets. There's no server required (other than the WebRTC signaling server), so it's private and secure too.

http://github.com/adhamsalama/sqlite-wasm-webrtc

Post reply on HN