141–150 of 229 posts
This is pretty neat! The calendar didn't work well for me. I could only seem to navigate by month. And when I selected the earliest day (after much tapping), nothing seemed to be updated. Nonetheless, random access history is cool.
This would be awesome as a cross platform app.
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
And I just now added a 'me' view. Enter your username and it will show your comments/posts on any day. So you can scrub back through your 2006 - 2025 retrospective using the calendar buttons.
Earlier quoted context omitted.
I love this so much, on my phone this is much faster than actual HN (I know it's only a read-only version). Where did you get the 22GB figure from? On the site it says: > 46,399,072 items, 1,637 shards, 8.5GB, spanning Oct 9, 2006 to Dec 28, 2025
> Where did you get the 22GB figure from? The HN post title (:
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
I am curios why they don't use a single file and HTTP Range Requests instead. PMTiles (a distribution of OpenStreetMap) uses that.
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
The GitHub page is no longer available, which is a shame because I'm really interested in how this works. How was the entirety of HN stored in a single SQLite database? In other words, how was the data acquired? And how does the page load instantly if there's 22GB of data having to be downloaded to the browser?
- 1. download_hn.sh - bash script that queries BigQuery and saves the data to *.json.gz
- 2. etl-hn.js - does the sharding and ID -> shard map, plus the user stats shards.
- 3. Then either npx serve docs or upload to CloudFlare Pages.
The ./toool/s/predeploy-checks.sh script basically runs the entire pipeline. You can do it unattended with AUTO_RUN=true
It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.
Fear not. I have a collaborative project designed to address this.
The only way you could theoretically extract everyone's comment scores (at least the top level ones) would be like this if you're a complete madman:
1. Wait 48 hours so the article is effectively dead
2. Post a new comment using an account called ThePresident
3. Create a swarm of a thousand shill user accounts called Voter1, Voter2, etc.
4. Use a single account at a time and upvote ThePresident
5. Recheck the page to see if ThePresident has moved above a user(s) post
6. Record the score for that user and assign it to the tracked story's history
7. Repeat from (4)
Earlier quoted context omitted.
Fear not. I have a collaborative project designed to address this.
They're referring to scores on individual COMMENTS - this information isn't available via the HN Firebase API. The only way you could theoretically extract everyone's comment scores (at least the top level ones) would be like this if you're a complete madman: 1. Wait 48 hours so the article is effectively dead 2. Post a new comment using an account called ThePresident 3. Create a swarm of a thousand shill user accoun…
But the idea I have is not like that at all - it's much nicer on everyone's ethics. Stay tuned! :)
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
A read-only VFS doing this can be really simple, with the right API… This is my VFS: https://github.com/ncruces/go-sqlite3/blob/main/vfs/readervf... And using it with range requests: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv... And having it work with a Zstandard compressed SQLite database, is one library away: https://pkg.go.dev/github.com/SaveTheRbtz/zstd-seekable-form...
It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/
and why do you want wikipedia in your pocket, but not a smartphone? where do you draw the line?
(doing a lot of work in that area, so i am asking to learn from someone who might think alike)