https://hnify.com/leaderboard.html using the dataset tool too, amazing to have so much data freely available to play with.
Hacker News BigQuery Dataset
21–30 of 38 posts
Re: Hacker News BigQuery Dataset
#22Hi, Felipe Hoffa at Google here. We're aware the dataset hasn't been updated since a month ago, and we are working to fix it. You can track the issue here: - https://issuetracker.google.com/issues/127132286 In the meantime you can still play with the dataset, and dig into the full history of Hacker News - less this last month. I left some interesting queries to get you started here: - https://medium.com/@hoffa/hacker…
How do you run the import? Love to read more about how you consume the data
I would also like to know where the data comes from!
Re: Hacker News BigQuery Dataset
#23I’m actually fairly excited to learn about this. I painstakingly scrapped HN to build:
I’m excited about this alternative
Re: Hacker News BigQuery Dataset
#24Last year I built a domain leaderboard based on this dataset: https://hnleaderboard.com — planning to update for 2019 soon!
Re: Hacker News BigQuery Dataset
#25A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?
I support this question. Any comments ? NB: That's easy to downvote without commenting...
The Internet is written in ink. You should assume that any and all public posts you make have already been replicated and archived by countless parties in countless ways by the time you hit delete. HN public postings are no different.
The HN API [1] has been around in various forms for years and includes the same public data that's used to generate the public pages on the HN site, but rather than returning HTML pages designed for human consumption, the API returns the data in a JSON serialized form [2] designed for machine consumption [3].
When the HN API went live, it reduced the overhead and redundant work from all the programmers having to independently crawl and parse site. The HN BigQuery dataset is the same data returned by the HN API, Google just took the next step and did the work of loading it into BigQuery.
[1] https://github.com/HackerNews/API
[2] https://en.wikipedia.org/wiki/Category:Data_serialization_fo...
Re: Hacker News BigQuery Dataset
#26I added a simple api endpoint to access favorites on HN, since they weren’t available on the normal api.
Re: Hacker News BigQuery Dataset
#27Hi, Felipe Hoffa at Google here. We're aware the dataset hasn't been updated since a month ago, and we are working to fix it. You can track the issue here: - https://issuetracker.google.com/issues/127132286 In the meantime you can still play with the dataset, and dig into the full history of Hacker News - less this last month. I left some interesting queries to get you started here: - https://medium.com/@hoffa/hacker…
Any particular reason you don’t include user profiles in the dataset ? I ended up pulling them myself using the api...
Re: Hacker News BigQuery Dataset
#28A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?
If people are sharing their own PII in HN comments, they agreed to HN's T&Cs when signing up. Such T&Cs state (heavily trimmed for length): By uploading any User Content you hereby grant [..] a nonexclusive, worldwide [..] irrevocable license to [..] distribute [..] your User Content for any Y Combinator-related purpose in any form [..] Agreeing to the T&Cs and deliberately sharing information publicly covers the GDP…
The GDPR's super complicated, but I'm pretty sure its Right of Erasure, and specifically Article 7(3), which gives data subjects the right to withdraw consent at any time and the clause “it shall be as easy to withdraw consent as to give it" trumps any ridiculous "irrevocable" license to distribute your content in any form forever.
Also importantly, the GDPR requires that a controller not make a service conditional upon consent. Hacker News is likely not in compliance unless they make such data processing optional and require anyone interested to explicitly opt in.
But, then again, I'm not a lawyer, and even if I were, actual lawyers don't seen to know what the hell the GDPR actually requires either.
Re: Hacker News BigQuery Dataset
#29A dataset like this is going to have a bunch of personal information in it. When it’s distributed like this, how does that jive with regulations like GDPR? If a HN user would like to delete all their comments, how would that request be forwarded to every user of this dataset?
If people are sharing their own PII in HN comments, they agreed to HN's T&Cs when signing up. Such T&Cs state (heavily trimmed for length): By uploading any User Content you hereby grant [..] a nonexclusive, worldwide [..] irrevocable license to [..] distribute [..] your User Content for any Y Combinator-related purpose in any form [..] Agreeing to the T&Cs and deliberately sharing information publicly covers the GDP…
I’m not really sure that matters. GDPR includes the right to withdraw consent and the right to erasure, unless there are specific legal reasons why data can not be erased. Requests for erasure should be forwarded to third parties who use the data.
Re: Hacker News BigQuery Dataset
#30Earlier quoted context omitted.
If people are sharing their own PII in HN comments, they agreed to HN's T&Cs when signing up. Such T&Cs state (heavily trimmed for length): By uploading any User Content you hereby grant [..] a nonexclusive, worldwide [..] irrevocable license to [..] distribute [..] your User Content for any Y Combinator-related purpose in any form [..] Agreeing to the T&Cs and deliberately sharing information publicly covers the GDP…
The GDPR's super complicated, but I'm pretty sure its Right of Erasure, and specifically Article 7(3), which gives data subjects the right to withdraw consent at any time and the clause “it shall be as easy to withdraw consent as to give it" trumps any ridiculous "irrevocable" license to distribute your content in any form forever. Also importantly, the GDPR requires that a controller not make a service conditional u…
IAANAL, and I don't mean to single you out here, but this seemingly rational argument strikes me as subtle FUD. It's the type of argument that someone with a vested interest in collecting user data for profit might put forth in the hopes of polarizing the tech community and painting GDPR as out of touch with technical common sense.
Again, I'm not accusing you of anything here, I'm just pointing out who benefits from framing the conversation this way. So far there is a lot of precedent for small operators shutting down their sites out of fear of GDPR, but there is actually no precedent for regulators having actually gone after small operators for anything resembling reasonable practices. The day may come where EU regulators try to crack down on forums for who are unwilling or unable to redact users messages post-facto, but we're nowhere close to that today and I don't see strong reason to believe that's where we're headed either.