If anyone is working with the data, we should see how many users actually hivemind comment on things vs OC
A dataset of every Reddit comment
11–20 of 96 posts
Re: A dataset of every Reddit comment
#12There was also someone that created a dump of all HN data a while ago: https://github.com/sytelus/HackerNewsData
I have a GitHub repository for getting all the HN data the hard way using Python and the Algolia API, and storing them in PostgreSQL: https://github.com/minimaxir/get-all-hacker-news-submissions...
Re: A dataset of every Reddit comment
#13Anyone know how big it is yet?
Dataset on BigQuery is 265GB.
Re: A dataset of every Reddit comment
#14Anyone know how big it is yet?
160GB.
One bz2 file of comments per month.
Re: A dataset of every Reddit comment
#15Someone should try make a sentient twitter bot that learns from the Reddit data
Re: A dataset of every Reddit comment
#16As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel...
BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.)
I can answer any other questions that people have.
Re: A dataset of every Reddit comment
#17If anyone is working with the data, we should see how many users actually hivemind comment on things vs OC
You can't determine what is OC from comments, only from submission titles.
But by sheer coincidence, I have made a chart comparing average scores for OC submissions vs. non-OC submissions a few months ago: https://www.reddit.com/r/dataisbeautiful/comments/2rv76z/oc_...
Re: A dataset of every Reddit comment
#18I think it's fascinating that if anyone uses this dataset to train an AI, some tiny piece of my personality might make a contribution.
Re: A dataset of every Reddit comment
#19Not trying to throw cold water on this...but are the Reddit execs OK with this? I mean, Twitter and Facebook would most likely issue takedowns for similar kinds of data dumps...but is there something in the Reddit API TOS that says this is OK?...I wouldn't be surprised if there a fairly liberal license, as that would be aligned with the early spirit of the site. And an onerous TOS would have likely curbed the active user-run bot ecosystem that helps manage and monitor Reddit's discussions.
Re: A dataset of every Reddit comment
#20This could be amazing as input to a question answering engine.
I think Reddit was one of the places they had to wall off Watson from data-mining, because it devolved into foul-mouthed memes. I wish they hadn't, nothing would be better than a billion dollar piece of technology deciding to reenact Sean Connery Jeopardy skits, on Jeopardy.
[2001 HAL voice]: I have a question about the penis mightier. Does it work?