Live data from Hacker News

Show HN: 22 GB of Hacker News in SQLite

hackerbook.dosaygo.com

181–190 of 229 posts

Re: Show HN: 22 GB of Hacker News in SQLite

#181

Earlier quoted context omitted.

If the last story on HN was at December 26, that is.

Continuously updated != instantly updated

Continuously would suggest to me that the data is never far out of date, and a few days might be considered far in this case.

Perhaps “regularly updated” would be less contentious wordage?

Re: Show HN: 22 GB of Hacker News in SQLite

#182

Is it a thing that the design is almost unusable on a mobile phone? The tech making this possible is beyond cool, but it's just presented in such a brutal way for phone users, even though fixing it would be super simple.

Really? Let me know how I can help. What would you like to see fixed?

Re: Show HN: 22 GB of Hacker News in SQLite

#183

threw some heatmaps together of post volume and average score by day and time (15min intervals) story volume (all time): https://ibb.co/pBTTRznP average score (all time): https://ibb.co/KcvVjx8p story volume (since 2020): https://ibb.co/cKC5d7Pp average score (since 2020): https://ibb.co/WpN20kfh

added median too. median score (all time): https://ibb.co/gZV5QVMG median score (since 2020): https://ibb.co/Gfv8T7k8

This is fascinating, I love this! What do you create these heat maps with? Did you use the BigQuery data or download from the site?

Re: Show HN: 22 GB of Hacker News in SQLite

#184
post #68

Earlier quoted context omitted.

PMTiles is absurdly great software.

My only gripe is that the tile metadata is stored as JSON, which I get is for compatibility reasons with existing software, but for e.g. a simple C program to implement the full spec you need to ship a JSON parser on top of the PMTiles parser itself.

A JSON parser is less than a thousand lines of code.

Re: Show HN: 22 GB of Hacker News in SQLite

#185
post #162

Similar in spirit to a recent tool I recently posted Show HN on, https://exopriors.com/scry . You can use Claude Code to SQL+vector query HackerNews and many other high quality public commons sites, exceptionally well-indexed and usually 5+ minute query timeout limits, so you can run seriously large research queries, to rapidly refine your worldview (particular because you can do easily to EXHAUSTIVE exploration).

This looks cool but can you make a "Google Search Box" page where I don't have to sign in but can use it? It's just a bit of friction and I feel unbothered to overcome it. It's not personal to you - it's just how I feel about anything that looks unknown and interesting I just want to try, not have to sign up. For now. You know?

Re: Show HN: 22 GB of Hacker News in SQLite

#186

Earlier quoted context omitted.

added median too. median score (all time): https://ibb.co/gZV5QVMG median score (since 2020): https://ibb.co/Gfv8T7k8

This is fascinating, I love this! What do you create these heat maps with? Did you use the BigQuery data or download from the site?

thanks! i downloaded them and used python to make these

Re: Show HN: 22 GB of Hacker News in SQLite

#188

Earlier quoted context omitted.

This is fascinating, I love this! What do you create these heat maps with? Did you use the BigQuery data or download from the site?

thanks! i downloaded them and used python to make these

So cool! Would it be impossible fro me to use them on the Archive stats page (https://hackerbook.dosaygo.com/?view=archive) ? If you're okay with that any links/credit line details?

Totally cool if not, just super interesting!

Re: Show HN: 22 GB of Hacker News in SQLite

#189
post #22

Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…

Thanks! I'm glad you enjoyed the sausage being made. There's a little easter egg if you click on the compact disc icon. And I just now added a 'me' view. Enter your username and it will show your comments/posts on any day. So you can scrub back through your 2006 - 2025 retrospective using the calendar buttons.

I almost got tricked into trying to figure out what was Easter eggy about August 9 2015 :-) There's a clarifying tooltip on the link, but it is mostly obscured by the image's "Archive" title attribute.

Re: Show HN: 22 GB of Hacker News in SQLite

#190

Earlier quoted context omitted.

Thanks! I'm glad you enjoyed the sausage being made. There's a little easter egg if you click on the compact disc icon. And I just now added a 'me' view. Enter your username and it will show your comments/posts on any day. So you can scrub back through your 2006 - 2025 retrospective using the calendar buttons.

I almost got tricked into trying to figure out what was Easter eggy about August 9 2015 :-) There's a clarifying tooltip on the link, but it is mostly obscured by the image's "Archive" title attribute.

Oh, shit that was the problem! You solved the bug! I was trying to figure out why the right tooltip didn't display. A linked wrapped in an image wrapped in an easter egg! Or something. Ha, thank you. Will fix :)

edit: Fixed! Also I just pushed a new version with a Dec 29th Data Dump, so ... updates - yay!

Post reply on HN