121–130 of 229 posts
Is this updated regularly? 404 on GitHub as the other comment. With all due respect it would be great if there is an official HN public dump available (and not requiring stuff such as BigQuery which is expensive).
How much space is needed? ...for the data .... Im wondering if it would work on a tablet? ....
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
It'd be great if you could add it to Kiwix[1] somehow (not sure what the process is for that but 100rabbits figured it out for their site) - I use it all the time now that I have a dumb phone - I have the entirety of wikipedia, wiktionary and 100rabbits all offline. https://kiwix.org/en/
Suddenly occurs to me that it would be neat to pair a small LLM (3-7B) with an HN dataset
Looks like the repo was taken down (404). That's too bad, I'd like to see the inner-working with a subset of data, even with placeholders for the posts and comments.
Don't miss how this works. It's not a server-side application - this code runs entirely in your browser using SQLite compiled to WASM, but rather than fetching a full 22GB database it instead uses a clever hack that retrieves just "shards" of the SQLite database needed for the page you are viewing. I watched it in the browser network panel and saw it fetch: https://hackerbook.dosaygo.com/static-shards/shard_1636.sqli…
Is there anything more production grade built around the same idea of HTTP range requests like that sqlite thing? This has so much potential
I believe that there are also indexing opportunities (not necessarily via eg hive partitioning) but frankly - am kinda out of my depth pn it.
It's really a shame that comment scores are hidden forever. Would the admins consider publishing them after stories are old enough that voting is closed? It would be great to have them for archives and search indices and projects like this.
"1. Delayed Karma Display. I understand why comment karma was hidden. I don't see the harm in un-hiding karma after some time. If not 24 hours, then 72-168 hours. This would help me read through threads with 1300 comments."
This was last January. While I asked for a few more features, it is the only one that seems essential as HN grows with massive threads.
That repo is throwing up a 404 for me. Question - did you consider tradeoffs between duckdb (or other columnar stores) and SQLite?