Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hosting SQLite databases on GitHub Pages or any static file hoster

phiresky.github.io

Post reply on HN
Print view View original

231–240 of 252 posts

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#231

Post by crazygringo »

anonydsfsfs wrote: post #137
crazygringo wrote: post #89

This is hilariously clever. Using the "Range" HTTP header to read chunks of the database file absolutely works! But to be clear, there's no write equivalent, is there? You can't use "Range" with a PUT request.

The write equivalent would be the PATCH method using a "message/byteranges" body: https://tools.ietf.org/id/draft-wright-http-partial-upload-0...

Wow that's fascinating, thanks. That would actually turn HTTP into a kind of random-access filesystem, if adopted.

It's amazing but also slightly terrifying in the knowledge that then someone's going to write an SMB-over-HTTP connector.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#232

Post by crazygringo »

altfredd wrote: post #151
crazygringo wrote: post #87

Earlier quoted context omitted.

Not all webservers support/enable it, so YMMV. But as long as you're dealing with a known server that does, then gravy!

> Not all webservers support/enable it Could you provide an example of server that does not? AFAIK, Range is supported by all major CDNs, so not supporting it in web server would be a death knell for it's real-world adoption.

I can't think of a specific one, but a decent proportion (maybe a quarter?) of HTTP downloads I attempt don't support resuming a partial download, therefore don't support Range. (I.e. resuming always starts from the beginning.)

I would assume this is often because the site in question isn't using Apache etc. to serve a file directly, but is either essentially proxying it to some custom-built file serving service, or a script that processes/authenticates the file in some way, and they just never bothered to implement Range.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#233

Post by kelnos »

pjc50 wrote: post #90
phiresky wrote: post #85

Earlier quoted context omitted.

That's true, but it also means that random access will always use at least that amount of data even if it only has to fetch a tiny amount. I did a few (non-scientific) benchmarks on a few queries and 1kB seemed like an OK compromise. And note that the request chunk size is bound to the SQLite page size, and to change that page size you have to rewrite the whole DB. So it can't be set on the fly unless you have multip…

1kb fits in most IP MTU sizes, so that seems reasonable.

Do most HTTP responses have less than ~500 bytes of headers? I guess specifically here, GH pages' responses.

It looks like one of the requests made to the DB included a little over 700 bytes of response status line and headers, so that would probably end up spilling into more than one response packet, unfortunately.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#234

Post by f430 »

phiresky wrote: post #216
graderjs wrote: post #175

Man you are a frickin genius, seriously. like how you put all this together all the depth of knowledge of different topics this would require the low level and the high level and the way you explain it simply confidently and with impact. Your work is really an inspiration. You computer scienced the sheet out of this thing. this achievement, and this blog post, to me is on par with blog posts that you would see from a…

Thank you, I really appreciate it. It's pretty fun to do this kind of thing for yourself, but it's really rewarding to be able to share it with other people.

have you thought of getting this to work with network replicated p2p sqlite js library? forgot what it was called.

also does this mean that static sites now can scale without relying on db???

what other implications and application do you see?

wonderful work!!!! please let me know how i can donate

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#235

Post by M2Ys4U »

intergalplan wrote: post #224
nanna wrote: post #218

Earlier quoted context omitted.

Definitely. Just need to add a layer to the static site generator for it to populate the SQLite DB, right?

I'd also version the DB in the URL, else you could end up changing the file out from under someone who's already got the page loaded, with who-knows-what results depending on how different the file is. You could just prefix a head to every range request to check for changes, but that adds overhead and doesn't actually completely close the gap, so it'd still be possible to read a file different from the one you intend…

Requests could use the If-Unmodified-Since header to guard against DB changes without the overhead of a separate HEAD request.

If the underlying DB has been changed then the server should respond with 412 Precondition Failed.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#236

Post by tuukkah »

Cullinet wrote: post #146
havernator wrote: post #129

Earlier quoted context omitted.

TL;DR http, properly implemented, supports a ton more stuff than even many “web developers” are aware of, like… range requests, which are exactly what you’d think they’d be.

the most recent update to the W3C's own research webserver, written in Java, called Jigsaw, seems to be dated in 2007. I used it for a lot of purposes until 2002 but I don't know why I stopped working with Jigsaw only that by the time F# emerged in 2004 I was absorbed into a new direction : https://jigsaw.w3.org/ iirc Jigsaw was used to develop and validate the WebDAV protocols and XQUERY which at the time I remember…

The last link was a very interesting read - I wonder if BBC or anyone else has open-sourced a video streaming and editing system like that.

Connecting this to HTTP range requests, the edited video feed can consist of a list of the exact byte ranges that the clients need to download and play. Found this description of how Akamai uses range requests to serve low-latency streams: https://blogs.akamai.com/2020/11/using-ll-hls-with-byte-rang...

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#237

Post by e12e »

maskros wrote: post #226
IshKebab wrote: post #197

Earlier quoted context omitted.

Wow so it actually does something! I wish programs would use such vague descriptions. (Or more of them had helpful instant tooltips.)

Linearized (a.k.a. web optimized) PDF files only help for displaying the first page quickly. The rest of the file is still in pretty much random access order.

For what it's worth, there's a format for content like pdfs that's optimized for viewing on screens (as opposed to printing), djvu: https://en.m.wikipedia.org/wiki/DjVu

Oh, wow initial release 1998,now I'm feeling a bit old...

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#238

Post by e12e »

tlb wrote: post #74

The question I had is answered by this line of code: xhr.setRequestHeader("Range", "bytes=" + from + "-" + to); I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...

I guess "all" that's needed for write would be a webdav server with support for PATCH x-update-range? :)

https://tools.ietf.org/html/rfc5789

https://sabre.io/dav/http-patch/

Unfortunately, solid, stand-alone webdav servers are harder to come by than decent http2/1.1 servers.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#239

Post by wildrhythms »

batch12 wrote: post #214
Cullinet wrote: post #148

Earlier quoted context omitted.

BTW thank you havernator, because I have just realised what I can do with the setup I'm almost ready to pull the trigger on that'll give me a surfeit of online capacity (at least a baseload can be maintained while the rest is used for work instead of cloud time) : I am definitely going to investigate the possibility of providing a high level of standards specifications for simple web serving. If the W3C Jigsaw projec…

Not trying to offend, but this comment was hard to follow in a weird way. Along with your profile this makes me wonder-- are you gpt-2?

Their profile contains a full essay of near incomprehensible mumbo jumbo:

https://news.ycombinator.com/user?id=Cullinet

Post reply on HN