I have the feeling I'm watching some semi-major new web tech being born... out of the blue, in a blog post.
Fucking amazing, mad props. Beautiful work!
Hosting SQLite databases on GitHub Pages or any static file hoster
191–200 of 252 posts
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#192The question I had is answered by this line of code: xhr.setRequestHeader("Range", "bytes=" + from + "-" + to); I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...
One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.
I remember putting together a sloppy http range implementation that initially only supported single ranges, it had quite the explosive effect on adobe reader when it didn't get the expected response to its multi-range requests :)
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#193I wonder sometimes, what if we focus on producing native JS VM hardware, would it perform same as or better than WASM? Let's run everything on JS.
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#194Earlier quoted context omitted.
It's impressive on one hand. On the other it's still a lot of overhead.
For a casual or personal use case though, the alternative of running a client-server database on something like a VPS is probably more overhead than this. It's unlikely to be a very scalable option, but for use cases as described by the author it seems like a good fit.
I know of the drawbacks of the approach and wouldn't chose it for a lot of my projects, but I would say it is very scalable in those cases where I would. Put the DB on GitHub Pages for free along with your HTML/JS code and you can scale to whatever GitHub is willing and capable of delivering. Yes, your users might transfer way more data than needed but you pay nothing for it and do not have to maintain servers.
In the standard scenario for personal projects (not enterprise) I would have a small VPS/Dedicated server with a REST service running - that would be hugged to death immediately if a link would make it to some site like HN. And also, I completely share the experience of the Author that after a couple of years you have moved on, the VPS is dead etc and you don't want to invest time.
Again, before considering using solution, be sure to understand how it works and the resulting limitations or you will likely chose wrong.
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#195Earlier quoted context omitted.
Progressive JPEGs work well for this too, so you could have the same file used for a tiny thumbnail and large preview and full sized photo by sending different range requests. However you need to know how many bytes to request. I'm surprised this isn't used on mobile browsers to lower data usage. I'm sure with a little research you could figure out what a good mapping from pixel size to byte size should be to give go…
A browser doesn't have enough information to use this optimization. At the point where it's about to request an image, it doesn't know how large the resource will be, whether it'll be a progressive JPEG, or even whether it'll be a JPEG at all. Making range requests blindly would probably be a net loss -- for every progressive JPEG that the browser managed to save some time on, it'd have to make follow-up requests for…
You are correct.
However, one could use this approach: download as usual, and in a streaming fashion process the data and if it's a progressive JPEG, you can close the connection before you have received everything; and then you can cache the prefix and later download the rest if needed.
Fast clients will just swallow the whole file, while slow clients would be able to benefit from it.
It wouldn't work for pipelined HTTP connections though without cancelling the whole pipeline, so maybe not a very practical solution given the performance benefit that already gives. And HTTP/2 maybe doesn't support cancelling a transfer either, so.. ?
Maybe a direct "Accept" or "Prefer" header to indicate that it's enough to send just something useful for an icon would be a more ideal solution, but it would require server-side support.
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#196Earlier quoted context omitted.
What do you think is the size of the average SQLite database?
Given their widespread uses on phones, the overall average is likely very small. KBs even. A more relevant question might be: what would the average size of SQLite databases for web type (or even this specific use case) applications. I don't know, but 10s or 100s of MBs might not be a bad guess.
However, would those 10s or 100s of MBs be something web sites would just like to share in their entirety to the client to query as they wish? At least many commercial services would prefer to keep most of the data secret and even gain insight from the access requests.
But for a more open or a hobbyist project where minimizing amounts of data transfer is less important than minimizing the amount of doing more work (server-side code), then this seems like a decent solution.
It is also worth reminding that this solution only practically works for read-only databases.
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#197Earlier quoted context omitted.
I'm surprised that works, iirc pdf isn't defined in order and can't be parsed streaming
I think that's what the "optimize for Web" checkbox does
Wow so it actually does something! I wish programs would use such vague descriptions. (Or more of them had helpful instant tooltips.)
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#198Earlier quoted context omitted.
Have you actually read the article? SQLite is unmodified, and thinks it runs on a virtual file system, which fetches file chunks via HTTP range headers. It's REALLY impressive that you only need to read 54 KB out of 700 MB, to fetch the records.
Do most static site hosters support range requests?
Generally yes. Because not having range support means you can't resume file downloads. Which is a pretty essential feature for a static file host.
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#199The innovation here is getting sql.js to use http and range requests for file access rather than all being in memory. I wonder when people using next.js will start using this for faster builds for larger static sites?
Microsoft Access Cloud Edition, basically?
Sort of. Access had a "Forms" feature that let you create basic GUIs on top of your database. Also, the OP's project is (currently) only providing a read-only view of the SQLite database. Adding write support is possible but will be far less impressive to the HN crowd because SQLITE_BUSY will rear its ugly head ;-)
Re: Hosting SQLite databases on GitHub Pages or any static file hoster
#200Hi, I'm working on SkySQL, maybe it's useful for you: https://github.com/upshot-tech/SkySQL