Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A dataset of every Reddit comment

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 96 posts

Re: A dataset of every Reddit comment

#31

Post by minimaxir »

daave wrote: post #28
minimaxir wrote: post #16

As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can ans…

> Since the full dataset is ~285GB, you only get 4 queries per month. That's only true if your 4 queries need to read every single column. One of the big advantages of BigQuery's column-oriented storage is that you only pay to read the columns that are actually needed to answer your query. For example, this query to extract the top 10 authors only cost me 19GB to run (and took 7.0s): SELECT author, COUNT(*) AS COUNT…

Good point.

Although if you're doing analysis on the body column itself, it'll use the majority proportion of the data set, of course.

Re: A dataset of every Reddit comment

#32

Post by hackcasual »

danso wrote: post #19

Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active…

I believe reddit comments are made under a CC license

Re: A dataset of every Reddit comment

#33

Post by mbell »

Wow, that is much, much, smaller than I thought it would be.

Anyone have a recent link on Reddit's infrastructure? Given the small size I would think it easily fits in memory so I'm a bit curious how they handle it.

Re: A dataset of every Reddit comment

#34

Post by Lerc »

TTPrograms wrote: post #29

This would be a really amazing way to make factually backed statements about the nature of Reddit for news sources given the recent publicity. I.e. the frequency of comments of a certain nature, typical karma scores for those comments, breakdown by subreddit etc.

You have to be careful determining the nature of a certain comment algorithmically.

I saw one analysis of subreddits that checked for negativity. /r/PathOfExile came in as one of the most negative subreddits, which could easily be turned into a narrative about gaming culture.

To a person familiar with the context it seems far more likely that game concepts were skewing the data. Discussions involved killing, physical damage, life leech, killing, dying etc. Not to mention creature names themselves Devourers, Plummeting Ursa, Miscreations,

Re: A dataset of every Reddit comment

#35

Post by HOLYCOWBATMAN »

danso wrote: post #19

Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active…

Beyonce's Publicist is on it!

Re: A dataset of every Reddit comment

#36

Post by mattrepl »

hackcasual wrote: post #32
danso wrote: post #19

Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active…

I believe reddit comments are made under a CC license

Closest thing to an answer I've found is from the licensing page: https://www.reddit.com/wiki/licensing

But it doesn't specifically address collection and distribution of API results as a dataset.

(From the page:)

A licensing agreement is required in order to:

* use the reddit API for commercial purposes. Use of the API is considered "commercial" if you are earning money from it, including via in-app advertising or in-app purchases. Open source use is generally considered non-commercial.

* use the reddit alien logo ("snoo") in your app or for its thumbnail. Any new apps you create must be approved as well before usage. The circular "r" logo is reserved solely for use by reddit, Inc.

* allow users to subscribe to reddit gold via in-app purchases. If your platform allows for it, we encourage you to work with us to make this happen. We see gold purchases as a way for you to help reddit and to give back to the reddit community.

Re: A dataset of every Reddit comment

#37

Post by Houshalter »

tspike wrote: post #6

This could be amazing as input to a question answering engine.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general.

I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching on what is possible with that data.

The scope of the discussions on reddit is huge. Despite a lot of jokes in the comments here, the quality of the comments on average, is pretty decent. And the metadata like subreddit and score are extremely useful for filtering it down more.

Re: A dataset of every Reddit comment

#39

Post by jchomali »

hackcasual wrote: post #32
danso wrote: post #19

Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active…

I believe reddit comments are made under a CC license

Not sure if the comments are under any license.

Re: A dataset of every Reddit comment

#40

Post by ilyaeck »

minimaxir wrote: post #16

As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can ans…

Is this data in a format that allows to recover threads? I.e., comment X was a response to comment Y ?
Post reply on HN