171–180 of 252 posts
Earlier quoted context omitted.
One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.
I'm surprised that works, iirc pdf isn't defined in order and can't be parsed streaming
Earlier quoted context omitted.
B-Tree indexes are designed to work like this, to require a low number of IO operations. The index contains pointers to other places in the index.
So the answer is “yes, it has to download the index first”? None of these comments answer my question.
Earlier quoted context omitted.
Do most static site hosters support range requests?
More interestingly, do reverse-proxies like Varnish / CDNs like Cloudflare support range requests? If so, do they fetch the whole content on the back, and then allow arbitrary range requests within the cached content on the front?
Might it be feasible to easily strip down SQLite so that it only compiles the parts for read-only use? The browser version is obviously somewhat read-only but that's because of the sandbox. I'm talking about excluding the code for CREATE, UPDATE, INSERT and everything else which is just for writing. The aim here would be to produce a significantly smaller WASM binary.
I'm guessing that the answer is no, there's no easy way of doing this without significant rewrites of SQLite's core, but... I can't be the only one to think of this, surely?
this achievement, and this blog post, to me is on par with blog posts that you would see from a major company where they solve some significant business critical technical challenge in-house. for example: the GitHub blog post about how they created their spinning globe of commits on their homepage, or a Netflix blog post of how they optimized their Network infrastructure to serve so many customers.
your work is truly incredible. You're next level of next level.
Earlier quoted context omitted.
This certainly does look like an interesting solution, I'd be keen to try it myself. However, just in case you didn't already know about Lunr ( https://lunrjs.com/ ), it is fairly commonly used to implement search on static websites. e.g. https://squidfunk.github.io/mkdocs-material/ There are of course other similar libraries too. EDIT: Whoops, just saw a few comments below Lunr is already mentioned.
yeah I have come across Lunr and maybe a couple of other things in my research and I think for blogging it'll work well. What I'm interested in finding out is what works for a larger static site, that won't require you to load the index up-front. I'm also curious about how this sqlite thing picks the correct range to load (haven't looked at the code) and what the worst case might be.
Indexes are usually btree, it finds what and where the relevant index is (probably from the schema queries), then goes trawling through the b-tree.
That’s how db engines normally work, they don’t liberally go through the entire db contents.
If there’s no index then it finds the start of the table and scans it sequentially (which is exactly what it sounds like).
I have a question. It's a 668.8MB database file. What does actually happen if the query has to scan 300 mb before finding the right answer? Wouldn't it be better to do the work up front and deliver the answers as static json files? Sure you loose the flexibility of dynamic queries, but do you really have that flexibility in non trivial cases (e.g. 300 mb search)?
This is really cool! I wonder what the restrictions are and if we would ever be able to write to a SQLite db like this in the future. This could push more to the front end without needing to write apis.
The main restriction is that the DB really needs well fitting indexes, otherwise querying is really slow and fetches a lot of data. Regarding writing: You could of course implement a writing API with POST requests for changing pages of the database - but then you would lose most of the benefits of this (not requiring any special kind of server). I also thought about implementing a kind of overlay filesystem, where ch…
Pop open the network pane in the browser tools and try running this SQL query for a demo of how clever this is: select country_code, long_name from wdi_country order by rowid desc limit 100 It fetches just 54.2KB of new data (across 49 small HTTP requests) to return 100 results - from a statically hosted database file that's 668.8MB! I have an animated GIF demo of this here: https://twitter.com/simonw/status/13889338…
mind blown. how is this possible???