Live data from Hacker News

A dataset of every Reddit comment

news.ycombinator.com

51–60 of 96 posts

Re: A dataset of every Reddit comment

#51

If anyone is working with the data, we should see how many users actually hivemind comment on things vs OC

You can't determine what is OC from comments, only from submission titles. But by sheer coincidence, I have made a chart comparing average scores for OC submissions vs. non-OC submissions a few months ago: https://www.reddit.com/r/dataisbeautiful/comments/2rv76z/oc_...

Hey there! I know you! Posts are coming soon!

Re: A dataset of every Reddit comment

#52
Could this be used to model the lifecycle of a Reddit post? What makes one thread get more votes and be replied to more often versus another similar thread posted at the same time. Is there a time when a comment is more likely to be voted for, based on wall clock or relative to the start of the thread. What topics are more likely to receive votes than others. Is there a minimum time between when reposts can be posted and still get upvotes. What are the most reposted things on Reddit. Why do people keep voting for reposts.

Re: A dataset of every Reddit comment

#53

Earlier quoted context omitted.

I believe reddit comments are made under a CC license

According to Reddit's User Agreement user content: "You retain the rights to your copyrighted content or information that you submit to reddit ('user content') except as described below." And the exceptions just state that Reddit has a perpetual irrevocable worldwide license. So it seems like there's no default license and others don't have any automatic rights to use the content. Does this assessment seem correct? I…

Good find, I think I got reddit confused with stackoverflow or wikipedia. If there's no clear assignment of copyright to api users, I would imagine that would be problematic for 3rd party app makers.

Re: A dataset of every Reddit comment

#54
post #6

This could be amazing as input to a question answering engine.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

If it's open source I'll love to take a look

Re: A dataset of every Reddit comment

#55
post #44
post #41

Earlier quoted context omitted.

It's not the hard to filter put the foul-mouthed content

But what about phrases with double meanings (aka: Fun With English): Children make nutritious snacks

Not necessarily an issue in the grand scheme of things.

Re: A dataset of every Reddit comment

#56
post #42

Earlier quoted context omitted.

I actually built an IRC bot that did this once. It searched reddit for your question, took the first result, and posted the top comment. It worked very well for certain kinds of questions. Especially if it came from the better subreddits like askscience, but even just reddit in general. I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching o…

I'd be very interested to learn more about your project and findings. Is that bot still alive?

I turned it back on for the time being. It can be found at irc.snoonet.org at #mybots

https://kiwiirc.com/client/irc.snoonet.org/mybots

EDIT: Reddit changed their api, give me a moment to fix it.

EDIT2: It works now!

Re: A dataset of every Reddit comment

#57

Earlier quoted context omitted.

According to Reddit's User Agreement user content: "You retain the rights to your copyrighted content or information that you submit to reddit ('user content') except as described below." And the exceptions just state that Reddit has a perpetual irrevocable worldwide license. So it seems like there's no default license and others don't have any automatic rights to use the content. Does this assessment seem correct? I…

Good find, I think I got reddit confused with stackoverflow or wikipedia. If there's no clear assignment of copyright to api users, I would imagine that would be problematic for 3rd party app makers.

Nit: I don't think copyright assignment is the correct term here. That refers to transferring the ownership of the root rights to the material, but you're just referring to granting specific rights under a license.

Re: A dataset of every Reddit comment

#58
post #42

Earlier quoted context omitted.

I'd be very interested to learn more about your project and findings. Is that bot still alive?

I turned it back on for the time being. It can be found at irc.snoonet.org at #mybots https://kiwiirc.com/client/irc.snoonet.org/mybots EDIT: Reddit changed their api, give me a moment to fix it. EDIT2: It works now!

AMAbot - is that the one?

Re: A dataset of every Reddit comment

#59
post #58

Earlier quoted context omitted.

I turned it back on for the time being. It can be found at irc.snoonet.org at #mybots https://kiwiirc.com/client/irc.snoonet.org/mybots EDIT: Reddit changed their api, give me a moment to fix it. EDIT2: It works now!

AMAbot - is that the one?

Yes. It automatically replies to anything you say in the chat on that channel.

Re: A dataset of every Reddit comment

#60

As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can ans…

I want all the url submissions in a given subreddit, but all I can find in the tables is "link_id". How do I map link_ids to urls?
Post reply on HN