Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hosting SQLite databases on GitHub Pages or any static file hoster

phiresky.github.io

Post reply on HN
Print view View original

161–170 of 252 posts

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#161

Post by banana_giraffe »

killingtime74 wrote: post #154
banana_giraffe wrote: post #99

In a similar vein, I've mentioned this before, but if you're doing Python stuff, you can use the apsw package (not the one in PyPi, though) to write a VFS layer that SQLite will use to read the database. I've used this for the same basic idea as this article, only letting me store SQLite databases in AWS's S3 that I can access with AWS APIs so they don't need to be public. It works well, though it's absolutely not fo…

This one? https://rogerbinns.github.io/apsw/

Yep, exactly that one. There's a simple example of a VFS implementation on the examples page that's a reasonable starting point:

https://rogerbinns.github.io/apsw/example.html

Once you wrap your head around how you need to pass parameters to the helper, it's really straightforward, you just need to implement the xOpen and xRead calls.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#162

Post by cozzyd »

jiofih wrote: post #144
cozzyd wrote: post #63

If you just want static data, using jsroot or jsfive might be a better option.

I think you missed the main part: the dataset is not in memory. This is not SQLite-in-the-browser but using a virtual file system over HTTP. Neither of those alternatives do anything similar.

Same with those! They'll just grab the chunks they need (also with range requests) to do what you want from binary ROOT or (presumably, I haven't actually used jsfive) HDF5 files.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#163

Post by derefr »

throwawayboise wrote: post #102
watermelon0 wrote: post #50

Earlier quoted context omitted.

Have you actually read the article? SQLite is unmodified, and thinks it runs on a virtual file system, which fetches file chunks via HTTP range headers. It's REALLY impressive that you only need to read 54 KB out of 700 MB, to fetch the records.

Do most static site hosters support range requests?

More interestingly, do reverse-proxies like Varnish / CDNs like Cloudflare support range requests? If so, do they fetch the whole content on the back, and then allow arbitrary range requests within the cached content on the front?

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#164

Post by NoInkling »

crazygringo wrote: post #89

This is hilariously clever. Using the "Range" HTTP header to read chunks of the database file absolutely works! But to be clear, there's no write equivalent, is there? You can't use "Range" with a PUT request.

Even if there was, I can't imagine your everyday static host ever supporting it.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#165

Post by fy20 »

scottlamb wrote: post #150
fanf2 wrote: post #114

Earlier quoted context omitted.

One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.

Also .mp4 files. The format is designed for seekability, and browsers take advantage of this.

Progressive JPEGs work well for this too, so you could have the same file used for a tiny thumbnail and large preview and full sized photo by sending different range requests. However you need to know how many bytes to request.

I'm surprised this isn't used on mobile browsers to lower data usage. I'm sure with a little research you could figure out what a good mapping from pixel size to byte size should be to give good enough results.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#167

Post by throw123123123 »

altano wrote: post #160
city41 wrote: post #44

Earlier quoted context omitted.

I'm guessing they mean rather than build a static Next site that generates 10k+ pages (or whatever large means in the given context), it instead creates one page that just queries the data from the client. I have one Next static site that has about 20k pages and takes about 20 minutes to build and deploy. I think that's an acceptable build time. But I do know of other people around the net who have mentioned having s…

For really large sites Next.js already has Incremental Static Regeneration which is usually the right solution to fast [re]builds: https://www.smashingmagazine.com/2021/04/incremental-static-...

Its not the same because you have to rebuild all the pages if you change your data source. In this implementation you can upload a new data set and it will work.

Its just a different stack.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#168

Post by sp332 »

scottlamb wrote: post #150
fanf2 wrote: post #114

Earlier quoted context omitted.

One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.

Also .mp4 files. The format is designed for seekability, and browsers take advantage of this.

You can even point VLC at a .iso file on a web server, and seek around in it.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#169

Post by enneff »

altfredd wrote: post #151
crazygringo wrote: post #87

Earlier quoted context omitted.

Not all webservers support/enable it, so YMMV. But as long as you're dealing with a known server that does, then gravy!

> Not all webservers support/enable it Could you provide an example of server that does not? AFAIK, Range is supported by all major CDNs, so not supporting it in web server would be a death knell for it's real-world adoption.

Depends on what’s being served. Any decent static file server should support it, but if the content is at all dynamically produced then the authors would have to think to implement it and rarely do.
Post reply on HN