Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Full Hacker News dataset now available on BigQuery

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 48 posts

Tell HN: Full Hacker News dataset now available on BigQuery

#1

Post by minimaxir »

https://bigquery.cloud.google.com/table/fh-bigquery:hackernews.comments

The dataset is up-to-date for October 2015 and uses the official HN API as a data source. (Unfortuntely, this includes the HTML formatting in the comments) Dataset is about 4GB total; since BigQuery allows for 1000GB processing for free each month, it is effectively no cost to analyze.

Felipe Hoffa (who uploaded the dataset) has links to the dataset along with some sample Python code for analysis: https://github.com/fhoffa/notebooks/blob/master/analyzing%20hacker%20news.ipynb

I have a few scripts for downloading all the data manually (https://github.com/minimaxir/get-all-hacker-news-submissions-comments ), but the BigQuery table may be more pragmatic. I can do some more sample queries if requested. (See my BigQuery tutorial for Reddit data: http://minimaxir.com/2015/10/reddit-bigquery/ )

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#3

Post by nsmalch »

Nice trove to pore through when I find the time.

I like to use Twitter to analyze HN datasets. It's mostly limited to links, because that's what I'm after mostly.

https://twitter.com/newsycombinator https://twitter.com/HackerNews .. And a few other accounts. Try to avoid Bitly wrapped links.

Use something like Greptweet to harvest the tweets and parse out any noise.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#4

Post by minimaxir »

nsmalch wrote: post #3

Nice trove to pore through when I find the time. I like to use Twitter to analyze HN datasets. It's mostly limited to links, because that's what I'm after mostly. https://twitter.com/newsycombinator https://twitter.com/HackerNews .. And a few other accounts. Try to avoid Bitly wrapped links. Use something like Greptweet to harvest the tweets and parse out any noise.

That will not get you accurate results for HN data analysis since a) those accounts only tweet important links so analysis will be biased b) you can only get 3200 tweets at a time. (This is a Twitter API limitation)

You have to look at both the good and the bad.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#6

Post by nsmalch »

minimaxir wrote: post #4
nsmalch wrote: post #3

Nice trove to pore through when I find the time. I like to use Twitter to analyze HN datasets. It's mostly limited to links, because that's what I'm after mostly. https://twitter.com/newsycombinator https://twitter.com/HackerNews .. And a few other accounts. Try to avoid Bitly wrapped links. Use something like Greptweet to harvest the tweets and parse out any noise.

That will not get you accurate results for HN data analysis since a) those accounts only tweet important links so analysis will be biased b) you can only get 3200 tweets at a time. (This is a Twitter API limitation) You have to look at both the good and the bad.

I hear you. Raw unfiltered links always have hidden gems.

One thing though: Greptweet has an archive somewhere with a huge trove of tweets that users of the service have searched for, and were thus logged and kept. (Some even go over the 3200 limit). It's a massive Tarball, so set aside time to download it and parse out boring/noisy links.

A lot of HN links are tech-press posts which consist of hearsay and merely proxy the thoughts of others. The recent changes in HN with regards to more academia-style posts is refreshing.

Post reply on HN