A dataset of every Reddit comment • Hacker News Board

Re: A dataset of every Reddit comment

#61

Post by Houshalter » Sun, Jul 12, 2015, 12:53 AM UTC

bl4ckdu5t wrote: post #54

Houshalter wrote: post #37

Earlier quoted context omitted.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

If it's open source I'll love to take a look

I pastebined the code. It's terrible. I should rewrite it. But here it is: http://pastebin.com/CM9u17jq

Re: A dataset of every Reddit comment

#62

Post by minimaxir » Sun, Jul 12, 2015, 1:02 AM UTC

rcpt wrote: post #60

minimaxir wrote: post #16

As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can ans…

I want all the url submissions in a given subreddit, but all I can find in the tables is "link_id". How do I map link_ids to urls?

There is no submission data yet.

/u/Stuck_in_the_Matrix commented on another thread that he's working on it.

Re: A dataset of every Reddit comment

#63

Post by minimaxir » Sun, Jul 12, 2015, 1:05 AM UTC

stuck_in_the_ma wrote: post #51

minimaxir wrote: post #17

Earlier quoted context omitted.

You can't determine what is OC from comments, only from submission titles. But by sheer coincidence, I have made a chart comparing average scores for OC submissions vs. non-OC submissions a few months ago: https://www.reddit.com/r/dataisbeautiful/comments/2rv76z/oc_...

Hey there! I know you! Posts are coming soon!

Good to hear! :)

Re: A dataset of every Reddit comment

#64

Post by linkmotif » Sun, Jul 12, 2015, 1:12 AM UTC

Does anyone know if this includes the spam? This dataset is obviously very interesting in its own right without spam, but far more useful for me would be all of it, including the spam.

EDIT: regardless, does anyone know if similar datasets with spam?

Re: A dataset of every Reddit comment

#65

Post by jedberg » Sun, Jul 12, 2015, 1:44 AM UTC

linkmotif wrote: post #64

Does anyone know if this includes the spam? This dataset is obviously very interesting in its own right without spam, but far more useful for me would be all of it, including the spam. EDIT: regardless, does anyone know if similar datasets with spam?

If they grabbed it via the API then no, it doesn't. It also won't include a lot of older stuff which is no longer accessible through the API.

Re: A dataset of every Reddit comment

#66

Post by colordrops » Sun, Jul 12, 2015, 1:52 AM UTC

Does this data include private subreddits?

Re: A dataset of every Reddit comment

#67

Post by Houshalter » Sun, Jul 12, 2015, 1:53 AM UTC

Houshalter wrote: post #37

tspike wrote: post #6

This could be amazing as input to a question answering engine.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

Sample of conversation with it. Not cherry picked, just showing some of the good and the bad:

https://i.imgur.com/LDD9isL.png?1

Re: A dataset of every Reddit comment

#68

Post by sdrothrock » Sun, Jul 12, 2015, 2:41 AM UTC

mbell wrote: post #33

Wow, that is much, much, smaller than I thought it would be. Anyone have a recent link on Reddit's infrastructure? Given the small size I would think it easily fits in memory so I'm a bit curious how they handle it.

This is the newest (2010) one I know of, and probably the one everyone's seen (I think it was HN back when it was first posted: http://highscalability.com/blog/2010/5/17/7-lessons-learned-...

Re: A dataset of every Reddit comment

#69

Post by Widdershin » Sun, Jul 12, 2015, 3:59 AM UTC

nathan_f77 wrote: post #18

I think it's fascinating that if anyone uses this dataset to train an AI, some tiny piece of my personality might make a contribution.

You might enjoy http://reddit.com/r/subredditsimulator, a markov chain powered simulation of Reddit, fueled by all the different subreddits.

Re: A dataset of every Reddit comment

#70

Post by x0 » Sun, Jul 12, 2015, 4:01 AM UTC

stuck_in_the_ma wrote: post #49

Houshalter wrote: post #38

I really want to get my full commenting history from this, but the download is enormous. Anyone know how I could go about this with cloud services?

Hey there! I'm the one that uploaded the original dataset. I'm creating API endpoints that will easily allow you to do this. It should be completed in 1-2 weeks.

Beautiful!