Live data from Hacker News

Hosting SQLite databases on GitHub Pages or any static file hoster

phiresky.github.io

141–150 of 252 posts

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#141

Earlier quoted context omitted.

Do you think your team’s usage of SQLite is representative of the average SQLite user?

This is the fundamental flaw of 80% thinking. The fact that SQLite continues to reach for more users is what has made it such a successful general-purpose tool.

You didn’t answer the question.

> The fact that SQLite continues to reach for more users is what has made it such a successful general-purpose tool.

I never disputed this. You’re responding to a straw man.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#142

Over high latency links this would be virtually unusable. Why not just download the entire database into memory over XHR on page load? SQLite databases of pure data usually aren’t over 10MB in size.

I have been using SQLite databases for a few user application that has been running for close to a decade now. They are usually about 1.5GB. BTW, SQLite has a (theoretical?) max size of 140TB! (or so I've read)

What do you think is the size of the average SQLite database?

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#144
post #63

If you just want static data, using jsroot or jsfive might be a better option.

I think you missed the main part: the dataset is not in memory. This is not SQLite-in-the-browser but using a virtual file system over HTTP. Neither of those alternatives do anything similar.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#145
post #106

Earlier quoted context omitted.

mind blown. how is this possible???

TL;DR http, properly implemented, supports a ton more stuff than even many “web developers” are aware of, like… range requests, which are exactly what you’d think they’d be.

Also SQLite store data in pages and the page size can be tweaked. Combined with range requests any part of the database can be requested.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#146
post #106

Earlier quoted context omitted.

mind blown. how is this possible???

TL;DR http, properly implemented, supports a ton more stuff than even many “web developers” are aware of, like… range requests, which are exactly what you’d think they’d be.

the most recent update to the W3C's own research webserver, written in Java, called Jigsaw, seems to be dated in 2007. I used it for a lot of purposes until 2002 but I don't know why I stopped working with Jigsaw only that by the time F# emerged in 2004 I was absorbed into a new direction :

https://jigsaw.w3.org/

iirc Jigsaw was used to develop and validate the WebDAV protocols and XQUERY which at the time I remember thinking XQUERY was sure to be the future as implementations of advanced data management and manipulation and query and distribution and declaration looked to be what the whole point of webservers were for. The incredible distractions caused by "rich media" as opposed to multimedia as it was understood then, are really worth thinking about. Saying that, however, the BBC is doing excellent work on restoring the balance of necessary powers to the network standards engineers https://www.bbc.co.uk/sounds/help/questions/about-bbc-sounds...

https://www.bbc.co.uk/rd/blog/2014-03-media-source-extension...

https://www.bbc.co.uk/rd/projects/nearly-live-production

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#148
post #106

Earlier quoted context omitted.

mind blown. how is this possible???

TL;DR http, properly implemented, supports a ton more stuff than even many “web developers” are aware of, like… range requests, which are exactly what you’d think they’d be.

BTW thank you havernator, because I have just realised what I can do with the setup I'm almost ready to pull the trigger on that'll give me a surfeit of online capacity (at least a baseload can be maintained while the rest is used for work instead of cloud time) : I am definitely going to investigate the possibility of providing a high level of standards specifications for simple web serving. If the W3C Jigsaw project had been maintained, I'd simply put it up and invite interested users to persuade me to send them a shell login. OK obviously that's far too naive today, but I would love to run a especially standards compliant host for negligible nominal or even no charge so people could maybe get a view of better ways to present the WWW.

frankly I think that unless we do things like this, the Internet is simply going to become a closed shop to anyone not wielding enterprise budgets and legal department capabilities.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#149
post #6

This is really cool! I wonder what the restrictions are and if we would ever be able to write to a SQLite db like this in the future. This could push more to the front end without needing to write apis.

The main restriction is that the DB really needs well fitting indexes, otherwise querying is really slow and fetches a lot of data. Regarding writing: You could of course implement a writing API with POST requests for changing pages of the database - but then you would lose most of the benefits of this (not requiring any special kind of server). I also thought about implementing a kind of overlay filesystem, where ch…

Seems like it might be possible to add WebRTC(or some other peer-based live system, I don't know the performance characteristics of WebRTC) to get realtime writes. Those edits would then be added to the WAL, and every say, 10 minutes anyone who has been active for 10 minutes could checkpoint and push the changes to github.

It's slightly more centralized than perfect, but man do you get a lot for a little.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#150
post #114
post #74

The question I had is answered by this line of code: xhr.setRequestHeader("Range", "bytes=" + from + "-" + to); I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...

One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.

Also .mp4 files. The format is designed for seekability, and browsers take advantage of this.
Post reply on HN