Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hosting SQLite databases on GitHub Pages or any static file hoster

phiresky.github.io

Post reply on HN
Print view View original

131–140 of 252 posts

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#131

Post by gawin »

nicoburns wrote: post #117
gitgud wrote: post #111

Earlier quoted context omitted.

Generating 20k pages in 20mins is impressive, 16 pages a second on average. In my experience, it can take a couple of minutes just to deploy 20 pages, but that could just be the overhead of Typescript and SASS compilation too...

Hugo claims <1ms a page. Which would mean 20k pages in under 20 seconds. 20k pages in 20 mins is not fast!

Confirmed: Build a 30k pages site with Hugo in 53 seconds, including grabbing JSON content from Contentful and deploy on AWS S3.

Took some effort to get everything under 60 seconds, speed was the main reason to use Hugo.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#132

Post by lsmurray »

nindalf wrote: post #70
city41 wrote: post #44

Earlier quoted context omitted.

I'm guessing they mean rather than build a static Next site that generates 10k+ pages (or whatever large means in the given context), it instead creates one page that just queries the data from the client. I have one Next static site that has about 20k pages and takes about 20 minutes to build and deploy. I think that's an acceptable build time. But I do know of other people around the net who have mentioned having s…

You should write a post about this if you implement it. My humble suggestion for blog post title - 'Speed up your Next.js builds with this author's one weird trick! Vercel hates him!'

I generate my entire blog on nextjs from a sqlite data base already. https://lsmurray.com/blog/database-first-development

It’s got a ton of rough edges but the boilerplate is there to get a proof of concept pretty quickly

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#134

Post by iudqnolq »

fanf2 wrote: post #114
tlb wrote: post #74

The question I had is answered by this line of code: xhr.setRequestHeader("Range", "bytes=" + from + "-" + to); I am a little surprised you can just do that. In https://github.com/phiresky/sql.js-httpvfs/blob/master/src/l...

One of the heaviest users of range requests is (or was) the Adobe Acrobat PDF plugin.

I'm surprised that works, iirc pdf isn't defined in order and can't be parsed streaming

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#135

Post by iudqnolq »

laszlokorte wrote: post #66
drtournier wrote: post #60

Solution works really well to databases which will not be updated frequently, like a standalone site. Although one should be aware of one very important git behavior - git does not diff binary files (like SQLite dbs). That means 2 things: 1. Each db update will generate a new file in git, maintaining the whole old file in history, instead of the diff in bytes. This will accumulate a lot of clutter in the repo 2. As g…

Minor nitpick: Git does not store diffs for any file format but always the full file for each version. So it does not really matter that its binary (except for not being able to VIEW the diff, but I guess you could even implement a plugin for that) but just that it's a big file. Even a huge text file would be fully stored per version. /edit: The sibling comments mentions that git can infact delta compress older commi…

This is mostly correct. Git's core object model is snapshots, which can then optionally be compressed. That should be transparent though.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#136

Post by nsonha »

Benjamin_Dobell wrote: post #130
nsonha wrote: post #127

This is perfect for my need, been looking for a way to add search to my static site completely free of server. Now I can use sqlite as index.

This certainly does look like an interesting solution, I'd be keen to try it myself. However, just in case you didn't already know about Lunr ( https://lunrjs.com/ ), it is fairly commonly used to implement search on static websites. e.g. https://squidfunk.github.io/mkdocs-material/ There are of course other similar libraries too. EDIT: Whoops, just saw a few comments below Lunr is already mentioned.

yeah I have come across Lunr and maybe a couple of other things in my research and I think for blogging it'll work well. What I'm interested in finding out is what works for a larger static site, that won't require you to load the index up-front. I'm also curious about how this sqlite thing picks the correct range to load (haven't looked at the code) and what the worst case might be.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#137

Post by anonydsfsfs »

crazygringo wrote: post #89

This is hilariously clever. Using the "Range" HTTP header to read chunks of the database file absolutely works! But to be clear, there's no write equivalent, is there? You can't use "Range" with a PUT request.

The write equivalent would be the PATCH method using a "message/byteranges" body: https://tools.ietf.org/id/draft-wright-http-partial-upload-0...

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#138

Post by brandmeyer »

CyberRabbi wrote: post #88
brandmeyer wrote: post #57

Earlier quoted context omitted.

My team has a few TB of data in SQLite files that are themselves dozens of GB each. We're using them as a replacement for leveldb's sstables, but with the structure of full SQL. It is highly effective.

Do you think your team’s usage of SQLite is representative of the average SQLite user?

This is the fundamental flaw of 80% thinking. The fact that SQLite continues to reach for more users is what has made it such a successful general-purpose tool.

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#139

Post by mythrwy »

CyberRabbi wrote: post #14

Over high latency links this would be virtually unusable. Why not just download the entire database into memory over XHR on page load? SQLite databases of pure data usually aren’t over 10MB in size.

I have been using SQLite databases for a few user application that has been running for close to a decade now. They are usually about 1.5GB.

BTW, SQLite has a (theoretical?) max size of 140TB! (or so I've read)

Re: Hosting SQLite databases on GitHub Pages or any static file hoster

#140

Post by durkie »

Valgrim wrote: post #122
ianlevesque wrote: post #16

This is easily the most clever web programming hack I’ve seen this year. Bravo. I had seen this used for video or audio of course but it never occurred to me you could use it for databases. There are probably a ton of other formats this is good for too.

I wonder if this could be used to serve dynamic maps.

I believe this is protomaps approach: re-encode the mbtiles (sqlite-based ) format in to something that can be requested with a http range request and thus served from a single dumb webserver that doesn't need to understand sqlite or mbtiles parsing
Post reply on HN