Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A dataset of every Reddit comment

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: A dataset of every Reddit comment

#81

Post by mcintyre1994 »

Houshalter wrote: post #67
Houshalter wrote: post #37

Earlier quoted context omitted.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

Sample of conversation with it. Not cherry picked, just showing some of the good and the bad: https://i.imgur.com/LDD9isL.png?1

This is awesome, I love the "current drama on reddit" answer..

Re: A dataset of every Reddit comment

#82

Post by shiggerino »

sneak wrote: post #30
thomasfromcdnjs wrote: post #15

Someone should try make a sentient twitter bot that learns from the Reddit data

People have been trying to make sentient bots of any kind for some time now.

I'd be extremely impressed with a somewhat general AI at the insect or nematode level, let alone human.

Re: A dataset of every Reddit comment

#83

Post by danellis »

mattrepl wrote: post #36
hackcasual wrote: post #32

Earlier quoted context omitted.

I believe reddit comments are made under a CC license

Closest thing to an answer I've found is from the licensing page: https://www.reddit.com/wiki/licensing But it doesn't specifically address collection and distribution of API results as a dataset. (From the page:) A licensing agreement is required in order to: * use the reddit API for commercial purposes. Use of the API is considered "commercial" if you are earning money from it, including via in-app advertising or i…

> use the reddit alien logo ("snoo") in your app or for its thumbnail

Do any of the apps actually do this, though? (Especially the free ones.)

Re: A dataset of every Reddit comment

#84

Post by anastasds »

danellis wrote: post #83
mattrepl wrote: post #36

Earlier quoted context omitted.

Closest thing to an answer I've found is from the licensing page: https://www.reddit.com/wiki/licensing But it doesn't specifically address collection and distribution of API results as a dataset. (From the page:) A licensing agreement is required in order to: * use the reddit API for commercial purposes. Use of the API is considered "commercial" if you are earning money from it, including via in-app advertising or i…

> use the reddit alien logo ("snoo") in your app or for its thumbnail Do any of the apps actually do this, though? (Especially the free ones.)

UReddit does, but may not be a great example because I received permission to use the logo in a private conversantion with hueypriest years ago (as long as I didn't try to monetize the project).

Re: A dataset of every Reddit comment

#86

Post by avinassh »

Houshalter wrote: post #67
Houshalter wrote: post #37

Earlier quoted context omitted.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

Sample of conversation with it. Not cherry picked, just showing some of the good and the bad: https://i.imgur.com/LDD9isL.png?1

awesome! how does one get started programming 'intelligent' things? I know how to program, haven't coded anything that is 'intelligent'. I would really love to know what all the things/concepts I should know to build a bot like yours. Thanks!

Re: A dataset of every Reddit comment

#88

Post by ytdht »

Houshalter wrote: post #38

I really want to get my full commenting history from this, but the download is enormous. Anyone know how I could go about this with cloud services?

here it is, only selected the body of your comments though, out of 4005 comments... http://s000.tinyupload.com/?file_id=09238386475544637092

here is the query that I used:

  SELECT
    author,
    body
    FROM
       [fh-bigquery:reddit_comments.2007],
       [fh-bigquery:reddit_comments.2008],
       [fh-bigquery:reddit_comments.2009],
       [fh-bigquery:reddit_comments.2010],
       [fh-bigquery:reddit_comments.2011],
       [fh-bigquery:reddit_comments.2012],
       [fh-bigquery:reddit_comments.2013],
       [fh-bigquery:reddit_comments.2014],
       [fh-bigquery:reddit_comments.2015_01],
       [fh-bigquery:reddit_comments.2015_02],
       [fh-bigquery:reddit_comments.2015_03],
       [fh-bigquery:reddit_comments.2015_04],
       [fh-bigquery:reddit_comments.2015_05]
   WHERE
       author = 'Houshalter'
this is where you can run another query: https://bigquery.cloud.google.com/table/fh-bigquery:reddit_c...

I did the same for my account and some comments are definitely missing... The first one that I cross-referenced is missing (5 month old and 1 point comment). But I have other comments with only 1 point and also 5 months old and those are there.

Re: A dataset of every Reddit comment

#89

Post by sova »

etblg wrote: post #71
alexggordon wrote: post #48

I used this[0] query to find the top ten[1] most downvoted comments of all time on Reddit. The most downvoted comment[2] is ironically in iAMA, by a mod of iAMA (ironic because of the recent drama). I'd find the top ten most upvoted, but I ran out of free bandwidth on BigQuery :(. [0] https://gist.github.com/alexggordon/7b56353dcf8044a7a5f9 [1] https://drive.google.com/file/d/0Bzxo-UKxFmN-eWticy1BR2tCRDQ... [2] https…

Here's the top ten most upvoted using that query with desc instead of asc: http://pastebin.com/Zvg6mdjZ

I don't even want to spoil any bit of the surprises that await you, dear readers. Thank you for BigQuerying this

Re: A dataset of every Reddit comment

#90

Post by Houshalter »

ytdht wrote: post #88
Houshalter wrote: post #38

I really want to get my full commenting history from this, but the download is enormous. Anyone know how I could go about this with cloud services?

here it is, only selected the body of your comments though, out of 4005 comments... http://s000.tinyupload.com/?file_id=09238386475544637092 here is the query that I used: SELECT author, body FROM [fh-bigquery:reddit_comments.2007], [fh-bigquery:reddit_comments.2008], [fh-bigquery:reddit_comments.2009], [fh-bigquery:reddit_comments.2010], [fh-bigquery:reddit_comments.2011], [fh-bigquery:reddit_comments.2012], [fh-big…

Thank you so much! I'd like to try it myself, but your link redirects me to some getting started page.

Any idea why some comments are missing? I haven't checked to see if they are all there, as far as I know they are.

I know that any in closed subreddits or comments that were removed by mods might be missing. But if you can access the comment's permalink without being signed in, then it should be in your data.

Post reply on HN