Tell HN: Full Hacker News dataset now available on BigQuery • Hacker News Board

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#41

Post by mintplant » Sat, Oct 24, 2015, 3:31 AM UTC

rndn wrote: post #29

fhoffa wrote: post #5

Hi! I'm the one that loaded this dataset into BigQuery. Feel free to ask any questions :). The notebook with sample queries and visualizations: https://github.com/fhoffa/notebooks/blob/master/analyzing%20...

There are items in there that are deleted/[dead] on HN (and not very recent). How come?

You can see dead things on HN if you turn on "showdead" in your account settings.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#42

Post by apryldelancey » Sat, Oct 24, 2015, 12:49 PM UTC

Thank you for this, I think it's great!

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#43

Post by arikfr » Sat, Oct 24, 2015, 5:19 PM UTC

If anyone wants to play with this dataset, and easily share his queries or visualize the data, you're welcome to use re:dash's demo instance [1]. I've also created an example query [2].

[1] http://demo.redash.io [2] http://demo.redash.io/queries/667/source#table

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#44

Post by manidoraisamy » Sat, Oct 24, 2015, 6:29 PM UTC

fhoffa wrote: post #5

Hi! I'm the one that loaded this dataset into BigQuery. Feel free to ask any questions :). The notebook with sample queries and visualizations: https://github.com/fhoffa/notebooks/blob/master/analyzing%20...

Great stuff! Looks like the whole dataset was imported 2 days back. If you can stream it daily, I can use table decorator and avoid processing all the records.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#45

Post by DanBC » Sat, Oct 24, 2015, 6:31 PM UTC

Cool.

I'd be interested to see a list of people who submit a lot. I submit too much - about one submission per day - and I'm curious what percentile that puts me in.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#46

Post by buzzsudo » Sat, Oct 24, 2015, 11:25 PM UTC

Hey guys check it out https://news.ycombinator.com/item?id=10445270

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#47

Post by arihant » Fri, Oct 30, 2015, 1:31 AM UTC

e12e wrote: post #40

beeboop wrote: post #37

Earlier quoted context omitted.

It's sort of a moot point. It's not especially difficult to scrape HN and compile a dataset like this for yourself.

It's a little like saying it's easy enough to make your own wine and sell it to minors. It is, but that doesn't make it legal.

No it's like that wine is available for everyone to buy anyway (via HTTP) and people are arguing that perhaps the bottled version (via non-HTTP protocol) should be age limited while keeping non-bottled version as is.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#48

Post by fhoffa » Fri, Oct 30, 2015, 9:58 PM UTC

cobookman wrote: post #32

All of the github event data is also on bigquery as well. https://www.githubarchive.org/#bigquery Now for someone to analyze github HN correlations.

I joined GitHub and Hacker News here - how many stars after a front page post?

https://www.reddit.com/r/bigquery/comments/3qpyor/joining_ha...