Live data from Hacker News

Hacker News BigQuery Dataset

console.cloud.google.com

11–20 of 38 posts

Re: Hacker News BigQuery Dataset

#11

Top Commentors of all time. tptacek is at 1st place with 33839 comments. Hacker news is 12 years old. That's an average of 7 comments per day since inception. Wow #standardSQL SELECT author, count(DISTINCT id) as `num_comments` FROM `bigquery-public-data.hacker_news.comments` WHERE id IS NOT NULL GROUP BY author ORDER BY num_comments DESC LIMIT 100;

Don't use the `comments` table: it was last updated December 2017.

On the full table:

    #standardSQL
    SELECT
     `by`,
     COUNT(DISTINCT id) as `num_comments`
    FROM `bigquery-public-data.hacker_news.full`
    WHERE id IS NOT NULL AND `by` != ''
    AND type='comment'
    GROUP BY 1
    ORDER BY num_comments DESC
    LIMIT 100
tptacek is in first place with 47283 comments.

Re: Hacker News BigQuery Dataset

#12
Hi, Felipe Hoffa at Google here.

We're aware the dataset hasn't been updated since a month ago, and we are working to fix it. You can track the issue here:

- https://issuetracker.google.com/issues/127132286

In the meantime you can still play with the dataset, and dig into the full history of Hacker News - less this last month. I left some interesting queries to get you started here:

- https://medium.com/@hoffa/hacker-news-on-bigquery-now-with-d...

Re: Hacker News BigQuery Dataset

#13

BigQuery keeps adding useless data. What we truly need is common crawl data then we can check specific site on our own. Or wait, BigQuery simply can't handle common crawl size dataset in their public service! Otherwise there is no reason to not add it, maybe it puts their search engine/ad business in geoparady. Is there any other Google public dataset BigQuery like platform? Where their direct search engine/ad platfo…

> Or wait, BigQuery simply can't handle common crawl size dataset in their public service!

This is not true. Source: ex-googler.

Re: Hacker News BigQuery Dataset

#14
post #8

A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?

The data is already publicly available for use and reuse, just in a different form. Why would it be any different than the rules regarding the public/api display of the information?

Re: Hacker News BigQuery Dataset

#15
post #14
post #8

A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?

The data is already publicly available for use and reuse, just in a different form. Why would it be any different than the rules regarding the public/api display of the information?

Maybe I’m misunderstanding what this is, is it not possible to query, download and process at a completely different scale than the API? If not, I suppose you might ask the same thing about the API.

Re: Hacker News BigQuery Dataset

#16
post #12

Hi, Felipe Hoffa at Google here. We're aware the dataset hasn't been updated since a month ago, and we are working to fix it. You can track the issue here: - https://issuetracker.google.com/issues/127132286 In the meantime you can still play with the dataset, and dig into the full history of Hacker News - less this last month. I left some interesting queries to get you started here: - https://medium.com/@hoffa/hacker…

Wow, what timing.

Late last night I had a conversation with someone explaining that Hacker News is not your typical message board -- it's owned and operated by YC and sits atop algorithms developed by some of the pioneers in spam and anomaly detection [1] [2], and it's is also an open dataset -- analyzed and scrutinized -- used by hackers worldwide to train and test bespoke AI.

HN is a live MNIST [3] for anomaly detection.

[1] http://www.paulgraham.com/spam.html

[2] http://googlesystem.blogspot.com/2007/07/paul-buchheit-man-b...

[3] http://yann.lecun.com/exdb/mnist/

Re: Hacker News BigQuery Dataset

#17
post #12

Hi, Felipe Hoffa at Google here. We're aware the dataset hasn't been updated since a month ago, and we are working to fix it. You can track the issue here: - https://issuetracker.google.com/issues/127132286 In the meantime you can still play with the dataset, and dig into the full history of Hacker News - less this last month. I left some interesting queries to get you started here: - https://medium.com/@hoffa/hacker…

How do you run the import? Love to read more about how you consume the data

Re: Hacker News BigQuery Dataset

#19
post #8

A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?

If people are sharing their own PII in HN comments, they agreed to HN's T&Cs when signing up. Such T&Cs state (heavily trimmed for length):

By uploading any User Content you hereby grant [..] a nonexclusive, worldwide [..] irrevocable license to [..] distribute [..] your User Content for any Y Combinator-related purpose in any form [..]

Agreeing to the T&Cs and deliberately sharing information publicly covers the GDPR's "consent" lawful base.

Even under GDPR this is not a situation where someone signed up for something else and then happen to have their personal data shared as a byproduct. They signed up to a site, agreed to T&Cs, and then explicitly and deliberately shared their personal data.

Post reply on HN