Downloading All of Hacker News Posts and Comments

Danke...on my 1,000th day (which was quite a few days ago), I wanted to do an analysis of how my upvoting/submission habits changed...that is, over the 3+ years I've been on HN, did my interests in hacking and languages diversify? But in terms of what I upvoted, it's hard to tell without the entire post set whether my interests changed, or the composition of HN's submissions.

Also, it's great to be able to filter through and find all of the highly-upvoted stories that I've missed out, and programmatically push them to Pinboard. Thanks for this.

Re: Downloading All of Hacker News Posts and Comments

#7

Post by meritt » Mon, Jun 02, 2014, 9:14 PM UTC

nekopa wrote: post #5

It's strange to see 1.3m stories and only 5.8m comments. I look forward to examining the data to see how many stories have 0 comments.

Lots and lots of stories never make it anywhere near the front page. Mostly spam.

Re: Downloading All of Hacker News Posts and Comments

#8

Post by minimaxir » Mon, Jun 02, 2014, 9:17 PM UTC

ivansavz wrote: post #2

Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hacker-news/ via https://news.ycombinator.com/item?id=7291531

I did, and with that, I had also included that Python code in GitHub, which uses the same implementation as the OP: https://github.com/minimaxir/hacker-news-download-all-storie...

Fun fact: At 10,000 calls/hour and 1,000 objects per request, you can download all stories AND comments in less than an hour.

As an aside: I tried to use ML on Hacker News stories and have had exactly zero success. (i.e. the predictive models are not statistically significantly better than the NIR)

Re: Downloading All of Hacker News Posts and Comments

#9

Post by nekopa » Mon, Jun 02, 2014, 9:17 PM UTC

meritt wrote: post #7

nekopa wrote: post #5

It's strange to see 1.3m stories and only 5.8m comments. I look forward to examining the data to see how many stories have 0 comments.

Lots and lots of stories never make it anywhere near the front page. Mostly spam.

But I think the spam is caught and doesn't register through the api...

Re: Downloading All of Hacker News Posts and Comments

#10

Post by nekopa » Mon, Jun 02, 2014, 9:21 PM UTC

howlett wrote: post #4

Your "Stories Download URL" is the same as the "Comments Download URL".

Easy fix, just change comments in the url to stories.