Live data from Hacker News

Full Reddit Submission Corpus now available for 2006 thru August 2015

reddit.com

61–70 of 100 posts

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#61
Last year, I did a statistical analysis with the same data and can confirm that the data is robust: http://minimaxir.com/2014/12/reddit-statistics/

The Comment data set is already on BigQuery, allowing for quick analysis without having to download that corpus (example: https://www.reddit.com/r/bigquery/comments/3kfnmq/reddit_sub... )

Once the Submission dataset is also on BigQuery, I'll write a blog post with more info on how to use it.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#62
post #60
post #25

Earlier quoted context omitted.

"More diverse" is, I believe, the term you're looking for. On reddit circa 2006, everybody had the same views on everything from politics to movies to cats to spiders. Eventually it acquired a more diverse cross-section of the population. Of course, they've been doing their best over the past few months to crack down on the expression of views they don't like, so I'm not sure what the demographic looks like nowadays…

Perhaps this is not the right thread for it, but what exactly about the Obama administration is idiotic? I'm ashamed to say I don't really follow politics.

Just one example is the "Cars for Clunkers" program, where the government bought old cars just to destroy them. It was supposed to be both an environmental and economic stimulus, but in reality was bad for the environment and is just a form of the Broken Window Fallacy.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#63
post #16

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

There sure as hell is lot more fear-mongering since the refugee "crisis" started.

3.7 million people that now are the largest refugee population in the world. You can leave off the quotes.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#64
post #51

Earlier quoted context omitted.

> Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. While I agree Reddit 2015 does reflect more of the US mainstream, it is still far from being an actual reflection of popular US opinion. Reddit is far more atheist/libertarian than the US mainstream. And I would argue it is right now far more reactionary/right wing as well. It has become very popular among young white…

> young white males who have unhealthy attitudes towards women due to the abundance of porn You sound just like the fundamentalist Evangelicals I grew up with.

It appears you didn't understand the sentence you are partially quoting. Reddit itself is full of porn. Much like attitudes towards women are likely to be generally more misogynistic among the patrons of a strip club than they are, say, inside of a planned parenthood, not just the presence but the omnipresence of porn on Reddit, and the role the porn plays in Reddit's popularity has an effect on the site and its users as a whole.

If you're going to claim that it has no effect, then you're delusional.

And my point is not that porn is bad or porn should be banned or whatever kind of crap you're trying to ascribe to me with your ridiculous comment. So, settle down there, Chester, and try actually reading what someone writes before responding to it. Try to actually comprehend the point and don't just react with the first knee-jerk thing that comes to mind.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#65
post #33

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

That chat bot is beyond amazing. Easily the best I've ever seen.

Me: What happens when an irresistible force meets and immovable object?

AMAbot: Too much explanation... Get out of the shower, you're a raisin. Also, makes me think of like babies, wrestling. Kind of like the opposite of atlas. why are you gay?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#66
post #64

Earlier quoted context omitted.

> young white males who have unhealthy attitudes towards women due to the abundance of porn You sound just like the fundamentalist Evangelicals I grew up with.

It appears you didn't understand the sentence you are partially quoting. Reddit itself is full of porn. Much like attitudes towards women are likely to be generally more misogynistic among the patrons of a strip club than they are, say, inside of a planned parenthood, not just the presence but the omnipresence of porn on Reddit, and the role the porn plays in Reddit's popularity has an effect on the site and its user…

Don't be too hard on him. As Ben Franklin said, the price of abundant, free porn is eternal vigilance.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#67
post #60

Earlier quoted context omitted.

Perhaps this is not the right thread for it, but what exactly about the Obama administration is idiotic? I'm ashamed to say I don't really follow politics.

Just one example is the "Cars for Clunkers" program, where the government bought old cars just to destroy them. It was supposed to be both an environmental and economic stimulus, but in reality was bad for the environment and is just a form of the Broken Window Fallacy.

It sold a ton of kia souls, and by my figgering probably got most people 10-20 extra miles per gallon.

Not that it makes up for the tragic deaths of many big engined classics....

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#68
post #58
post #52

Earlier quoted context omitted.

Limiting outgoing traffic would be enough.

Snowden used "a memory stick and other removable media, including a CD-ROM that he labeled as a Lady Gaga music CD" [1], so this kind of limiting wouldn't work. But he still did loads of internal network data queries, that's what I'm talking about. [1] http://www.wired.co.uk/news/archive/2013-06/14/snowden-memor...

You're mixing up Snowden with Manning.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#69
post #51
post #24

Earlier quoted context omitted.

You will most likely find a shift in the hive mind, but presumably only because Reddit grew from a smaller community with relatively dominant progressive and atheist/agnostic values into something much more mainstream. Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. So I'm not sure we can draw any conclusions from the hive mind shift.

> Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. While I agree Reddit 2015 does reflect more of the US mainstream, it is still far from being an actual reflection of popular US opinion. Reddit is far more atheist/libertarian than the US mainstream. And I would argue it is right now far more reactionary/right wing as well. It has become very popular among young white…

generally retrograde attitudes towards women of Reddit in general

That's a pretty awful accusation to level at ~20 million people. I wonder if you can substantiate that?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#70

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

This is super impressive.

My favorite was: "Who is your daddy and what does he do?" "It's not a tumor!"

Post reply on HN