Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Reddit Submission Corpus now available for 2006 thru August 2015

reddit.com

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#51

Post by ebbv »

socket0 wrote: post #24
TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

You will most likely find a shift in the hive mind, but presumably only because Reddit grew from a smaller community with relatively dominant progressive and atheist/agnostic values into something much more mainstream. Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. So I'm not sure we can draw any conclusions from the hive mind shift.

> Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer.

While I agree Reddit 2015 does reflect more of the US mainstream, it is still far from being an actual reflection of popular US opinion. Reddit is far more atheist/libertarian than the US mainstream. And I would argue it is right now far more reactionary/right wing as well. It has become very popular among young white males who have unhealthy attitudes towards women due to the abundance of porn and generally retrograde attitudes towards women of Reddit in general (both the users and the site itself.)

Same goes for its attitudes towards race.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#52

Post by h37680 »

r721 wrote: post #26
grey-area wrote: post #20

Earlier quoted context omitted.

But we cannot download that one. Not yet :) Given their lax security and general cluelessness of the people in charge, I'm quite sure it will leak at some point, at least partially, perhaps to corporations, perhaps to the public, just as internal NSA docs have leaked. It's very hard to keep things like that airtight forever - all it takes is one slip up and all the info stored could be accessible at some point in the…

I wonder whether they transformed top secret networks into intentionally slow networks now for large scale leaks to take years or tens of years :)

Limiting outgoing traffic would be enough.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#53

Post by netcraft »

grey-area wrote: post #12
lysium wrote: post #2

I'm not quite sure how I feel about it. Yes, that data was publicly available, but it 'feels' different that there is now an unalterable snapshot available to the public. No more comment deletion.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

What is this referring to?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#54

Post by bcruddy »

Houshalter wrote: post #11

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

This is really cool, is the source code available anywhere?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#55

Post by mason240 »

ebbv wrote: post #51
socket0 wrote: post #24

Earlier quoted context omitted.

You will most likely find a shift in the hive mind, but presumably only because Reddit grew from a smaller community with relatively dominant progressive and atheist/agnostic values into something much more mainstream. Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. So I'm not sure we can draw any conclusions from the hive mind shift.

> Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. While I agree Reddit 2015 does reflect more of the US mainstream, it is still far from being an actual reflection of popular US opinion. Reddit is far more atheist/libertarian than the US mainstream. And I would argue it is right now far more reactionary/right wing as well. It has become very popular among young white…

> young white males who have unhealthy attitudes towards women due to the abundance of porn

You sound just like the fundamentalist Evangelicals I grew up with.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#56

Post by pikzen »

netcraft wrote: post #53
grey-area wrote: post #12

Earlier quoted context omitted.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

What is this referring to?

NSA's Utah Data Center[0], where they store all the data siphoned off their various programs on the Internet.

[0] https://en.wikipedia.org/wiki/Utah_Data_Center

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#57

Post by grey-area »

netcraft wrote: post #53
grey-area wrote: post #12

Earlier quoted context omitted.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

What is this referring to?

Lots of state agencies collect info like Reddit comments as well as emails, and store it indefinitely. They're already quite good at linking online identities.

https://theintercept.com/2015/09/25/gchq-radio-porn-spies-tr...

https://en.m.wikipedia.org/wiki/Utah_Data_Center

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#58

Post by r721 »

h37680 wrote: post #52
r721 wrote: post #26

Earlier quoted context omitted.

I wonder whether they transformed top secret networks into intentionally slow networks now for large scale leaks to take years or tens of years :)

Limiting outgoing traffic would be enough.

Snowden used "a memory stick and other removable media, including a CD-ROM that he labeled as a Lady Gaga music CD" [1], so this kind of limiting wouldn't work. But he still did loads of internal network data queries, that's what I'm talking about.

[1] http://www.wired.co.uk/news/archive/2013-06/14/snowden-memor...

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#59

Post by runlevel1 »

Submissions Corpus Magnet Link:

    magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969
Comments Corpus Magnet Link

    magnet:?xt=urn:btih:32916ad30ce4c90ee4c47a95bd0075e44ac15dd2&dn=RC%5F2015-01.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969
Note: The comment corpus was built a few months ago (2015-01), so the two datasets don't overlap completely.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#60

Post by rifung »

hugh4 wrote: post #25
TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

"More diverse" is, I believe, the term you're looking for. On reddit circa 2006, everybody had the same views on everything from politics to movies to cats to spiders. Eventually it acquired a more diverse cross-section of the population. Of course, they've been doing their best over the past few months to crack down on the expression of views they don't like, so I'm not sure what the demographic looks like nowadays…

Perhaps this is not the right thread for it, but what exactly about the Obama administration is idiotic? I'm ashamed to say I don't really follow politics.
Post reply on HN