Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Full Hacker News dataset now available on BigQuery

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#31

Post by fhoffa »

catshirt wrote: post #27
fhoffa wrote: post #18

Earlier quoted context omitted.

Parent comment was deleted, but given the grandparent comment I'll guess you are asking how is BigQuery different to Redshift? There is an ongoing conversation about this on reddit: https://www.reddit.com/r/bigdata/comments/3jnam1/whats_your_...

the deleted comment asked how the BigQuery data set was different than http://hn.algolia.com

Bad guess then!

The answer is that the dataset must be very similar to the Algolia one, since both get their data from the same source: the Hacker News official API on Firebase.

(but Algolia keeps it up-to-date in realtime, while I haven't written anything to keep the BigQuery one updated - yet)

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#33

Post by fhoffa »

cobookman wrote: post #32

All of the github event data is also on bigquery as well. https://www.githubarchive.org/#bigquery Now for someone to analyze github HN correlations.

and reddit too

https://www.reddit.com/r/bigquery/comments/3cej2b/17_billion...

and wikipedia, and weather, and...

https://www.reddit.com/r/bigquery/wiki/datasets

:)

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#34

Post by [deleted] »

fhoffa wrote: post #31
catshirt wrote: post #27

Earlier quoted context omitted.

the deleted comment asked how the BigQuery data set was different than http://hn.algolia.com

Bad guess then! The answer is that the dataset must be very similar to the Algolia one, since both get their data from the same source: the Hacker News official API on Firebase. (but Algolia keeps it up-to-date in realtime, while I haven't written anything to keep the BigQuery one updated - yet)

[deleted]

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#35

Post by e12e »

arihant wrote: post #24
hoodoof wrote: post #8

Is this OK from a privacy perspective?

Except email, there is nothing private on HN to raise privacy issue. Public internet is written in ink.

Perhaps not in the content. But I'm not so sure there might not be actual legal issues (in jurisdiction that have privacy/data protection laws) with regards to stuff like being able to see patterns in login-times (based on post meta data), and similar. It's tenuous -- but just because things are a) on-line in the form of the main hn site, and b) accessible through the api, doesn't automatically mean that giving raw access to the data doesn't carry with it implications for whether or not the data should be considered personal/(more)sensitive.

I'm assuming this data set offers something the API doesn't (that's the point of uploading it, right? Other than just being an easy way for others to "back up" the hn data) -- and that can very well mean there are privacy/personal data implications.

I'm mostly stating this because of the principle, not because I think it's anything wrong with this concrete upload of hn data (and wrong != legal/illegal anyway).

As an example, at least in Norway, while you may have two separate databases that are linked to people, such as a database of cars passing through toll booths, and credit card charges from a large bar/restaurant chain -- it would generally be illegal to link the two.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#36

Post by malkia »

fhoffa wrote: post #16
rm999 wrote: post #14

Very cool, thanks! Looking forward to playing around with this. FYI, you named a column a reserved sql keyword ('by'). For future reference, and for others reading this: this is bad database design and makes it harder to use the table. You can get around this by wrapping the column name in brackets, like: >select ... where [by] = ...

I didn't name the column "by", I just gave the field the name that the API uses for it. And to make everyone's lives easier (including mine), I copied the [by] column to an [author] column, so you can do a >select ... where author = ... instead :)

Oh, first I thought there is some difference and did this:

  SELECT * FROM [fh-bigquery:hackernews.stories] WHERE [by] != author

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#37

Post by beeboop »

e12e wrote: post #35
arihant wrote: post #24

Earlier quoted context omitted.

Except email, there is nothing private on HN to raise privacy issue. Public internet is written in ink.

Perhaps not in the content . But I'm not so sure there might not be actual legal issues (in jurisdiction that have privacy/data protection laws) with regards to stuff like being able to see patterns in login-times (based on post meta data), and similar. It's tenuous -- but just because things are a) on-line in the form of the main hn site, and b) accessible through the api, doesn't automatically mean that giving raw…

It's sort of a moot point. It's not especially difficult to scrape HN and compile a dataset like this for yourself.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#38

Post by andreasvc »

0942v8653 wrote: post #9
hoodoof wrote: post #8

Is this OK from a privacy perspective?

Why wouldn't it be? All the data provided by the HN API is public.

Yes, when you submit a comment to HN you intend for it to be published on the HN site. However, using comments for text mining purposes is a different thing and in principle consent is required to do that. The default case is that user submitted content remains copyrighted by the author, with an implied consent for the site to publish the material on the site. There can be an explicit license such as with Wikipedia or StackOverflow; in that case text mining may be explicitly allowed.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#39

Post by andreasvc »

beeboop wrote: post #37
e12e wrote: post #35

Earlier quoted context omitted.

Perhaps not in the content . But I'm not so sure there might not be actual legal issues (in jurisdiction that have privacy/data protection laws) with regards to stuff like being able to see patterns in login-times (based on post meta data), and similar. It's tenuous -- but just because things are a) on-line in the form of the main hn site, and b) accessible through the api, doesn't automatically mean that giving raw…

It's sort of a moot point. It's not especially difficult to scrape HN and compile a dataset like this for yourself.

I don't see how difficulty has any bearing on privacy or copyright issues.

Re: Tell HN: Full Hacker News dataset now available on BigQuery

#40

Post by e12e »

beeboop wrote: post #37
e12e wrote: post #35

Earlier quoted context omitted.

Perhaps not in the content . But I'm not so sure there might not be actual legal issues (in jurisdiction that have privacy/data protection laws) with regards to stuff like being able to see patterns in login-times (based on post meta data), and similar. It's tenuous -- but just because things are a) on-line in the form of the main hn site, and b) accessible through the api, doesn't automatically mean that giving raw…

It's sort of a moot point. It's not especially difficult to scrape HN and compile a dataset like this for yourself.

It's a little like saying it's easy enough to make your own wine and sell it to minors. It is, but that doesn't make it legal.
Post reply on HN