181–190 of 252 posts
Earlier quoted context omitted.
It's impressive on one hand. On the other it's still a lot of overhead.
I would say it's less overhead than downloading the entire db to query it locally...? What is your suggestion for accessing a static database with less overhead?
Earlier quoted context omitted.
I believe this is protomaps approach: re-encode the mbtiles (sqlite-based ) format in to something that can be requested with a http range request and thus served from a single dumb webserver that doesn't need to understand sqlite or mbtiles parsing
This is the approach I took with http://github.com/protomaps/pmtiles , though it's optimized for the very specific use case of going from Z/X/Y integer coordinates to binary blobs, and takes shortcuts to accomplish that (fixed-width keys and root index page)
Doesn't the webserver have to seek from the beginning of the ~600mb file to the range you want, unless the file is in memory?
Earlier quoted context omitted.
Yeah, that was one of the inspirations for this. That one does not work in the browser though, would be a good project to do that same thing but with sqlite in wasm and integrated with WebTorrent instead of a native torrent program. I actually did also implement a similar thing fetching data on demand from WebTorrent (and in turn helping to host the data yourself by being on the website): https://phiresky.github.io/t…
This looks pretty efficient. Some chains can be interacted with without e.g. web3.js? LevelDB indexes aren't SQLite. Datasette is one application for views of read-only SQLite dbs with out-of-band replication. https://github.com/simonw/datasette There are a bunch of *-to-sqlite utilities in corresponding dogsheep project. Arrow JS for 'paged' browser client access to DuckDB might be possible and faster but without fu…
First of all, this is a very cool web hack, I like it very much. I have a question. It's a 668.8MB database file. What does actually happen if the query has to scan 300 mb before finding the right answer? Wouldn't it be better to do the work up front and deliver the answers as static json files? Sure you loose the flexibility of dynamic queries, but do you really have that flexibility in non trivial cases (e.g. 300 m…
Earlier quoted context omitted.
Also .mp4 files. The format is designed for seekability, and browsers take advantage of this.
Progressive JPEGs work well for this too, so you could have the same file used for a tiny thumbnail and large preview and full sized photo by sending different range requests. However you need to know how many bytes to request. I'm surprised this isn't used on mobile browsers to lower data usage. I'm sure with a little research you could figure out what a good mapping from pixel size to byte size should be to give go…
But as always, it's seldom the tools, but the right tool used for the wrong usecase that is the problem.
As everyone else has been saying, this is amazing work. It sounds like the biggest issue with loading the page is the initial sql.js download - it's about 1.2MB, is that right? Might it be feasible to easily strip down SQLite so that it only compiles the parts for read-only use? The browser version is obviously somewhat read-only but that's because of the sandbox. I'm talking about excluding the code for CREATE, UPDA…
Earlier quoted context omitted.
It's impressive on one hand. On the other it's still a lot of overhead.
I would say it's less overhead than downloading the entire db to query it locally...? What is your suggestion for accessing a static database with less overhead?