151–160 of 229 posts
link no workie: https://github.com/DOSAYGO-STUDIO/HackerBook
./toool/download-site.mjs --help
That way it's truly offline.
It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/
what dumb phone do you use? and why do you want wikipedia in your pocket, but not a smartphone? where do you draw the line? (doing a lot of work in that area, so i am asking to learn from someone who might think alike)
I have a $10 a month plan from US cellular with only 2gigs so I try to keep everything offline that I can.
Honestly it's mostly the news... so I draw the line at browser, I'll never install a browser, that's basically something I can do when I sit down at a PC. I read quite a bit and I like to have the ability to look up a word or a historical event or some reference from something I read using Kiwix and it's been great for that, just needed to add a 512gb micro sd card. And Libby I just use at the gym when I'm on the treadmill.
I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.
The hard-coded dictionary wouldn't be much stranger than Brotli's: https://news.ycombinator.com/item?id=27160590
Earlier quoted context omitted.
A read-only VFS doing this can be really simple, with the right API… This is my VFS: https://github.com/ncruces/go-sqlite3/blob/main/vfs/readervf... And using it with range requests: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv... And having it work with a Zstandard compressed SQLite database, is one library away: https://pkg.go.dev/github.com/SaveTheRbtz/zstd-seekable-form...
this does not caches the data right? it would always fetch from network? by any chance do you know of solution/extension that caches the data it would make it so much more efficient.
But, also, SQLite caches data; you can simply increase the page cache.
I tried "select * from items limit 10" and it is slowly iterating through the shards without returning. I got up to 60 shards before I stopped. Selecting just one shard makes that query return instantly. As mentioned elsewhere I think duckdb can work faster by only reading the part of a parquet file it needs over http. I was getting an error that the users and user_domains tables aren't available, but you just need t…
Doesn't `LIMIT` just limit the amount of rows returned, rather than the amount read & processed?
I wonder how much smaller it could get with some compression. You could probably encode "This website hijacks the scrollbar and I don't like it" comments into just a few bits.
I tried "select * from items limit 10" and it is slowly iterating through the shards without returning. I got up to 60 shards before I stopped. Selecting just one shard makes that query return instantly. As mentioned elsewhere I think duckdb can work faster by only reading the part of a parquet file it needs over http. I was getting an error that the users and user_domains tables aren't available, but you just need t…
That's odd. If it was a VFS, that's not what I'd expect would happen. Maybe it's not a VFS?
Earlier quoted context omitted.
The GitHub page is no longer available, which is a shame because I'm really interested in how this works. How was the entirety of HN stored in a single SQLite database? In other words, how was the data acquired? And how does the page load instantly if there's 22GB of data having to be downloaded to the browser?
You can see it now, forgot to make it public. - 1. download_hn.sh - bash script that queries BigQuery and saves the data to *.json.gz - 2. etl-hn.js - does the sharding and ID -> shard map, plus the user stats shards. - 3. Then either npx serve docs or upload to CloudFlare Pages. The ./toool/s/predeploy-checks.sh script basically runs the entire pipeline. You can do it unattended with AUTO_RUN=true
That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?
Maybe it got nuked by MS? The rest of their repo's are up.
BUT I did try to push the entire 10GB of shards to GitHub (no LFS, no thanks, money), and after the 20 minutes compressing objects etc, "remote hang up unexpectedly"
To be expected I guess. I did not think GH Pages would be able to do this. So have been repeating:
wrangler pages deploy docs --project-name static-news --commit-dirty=true