Full Reddit Submission Corpus now available for 2006 thru August 2015 • Hacker News Board

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#21

Post by scintill76 » Mon, Sep 28, 2015, 8:50 AM UTC

creshal wrote: post #16

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

There sure as hell is lot more fear-mongering since the refugee "crisis" started.

Why quotes on crisis? I'm not challenging it, because I also find it weird that overnight the phrase started popping up in the news daily. And what is the fear-mongering going on at Reddit?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#22

Post by raverbashing » Mon, Sep 28, 2015, 8:54 AM UTC

cmdli wrote: post #7

dssddsds wrote: post #6

Now we wait for someone to train a RNN and produce a fake but believable reddit parody with generated titles :)

Have you seen this? https://www.reddit.com/r/subredditsimulator

Yes, but this works with Markov chains, not RNNs

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#23

Post by TorKlingberg » Mon, Sep 28, 2015, 8:59 AM UTC

creshal wrote: post #16

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

There sure as hell is lot more fear-mongering since the refugee "crisis" started.

I think American redditors just don't care much, and there has been a lot of racism in European parts of Reddit for some time.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#24

Post by socket0 » Mon, Sep 28, 2015, 9:12 AM UTC

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

You will most likely find a shift in the hive mind, but presumably only because Reddit grew from a smaller community with relatively dominant progressive and atheist/agnostic values into something much more mainstream. Reddit 2006 would not reflect mainstream US values and opinions, Reddit 2015 would be much closer. So I'm not sure we can draw any conclusions from the hive mind shift.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#25

Post by hugh4 » Mon, Sep 28, 2015, 9:19 AM UTC

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

"More diverse" is, I believe, the term you're looking for.

On reddit circa 2006, everybody had the same views on everything from politics to movies to cats to spiders.

Eventually it acquired a more diverse cross-section of the population. Of course, they've been doing their best over the past few months to crack down on the expression of views they don't like, so I'm not sure what the demographic looks like nowadays and how many people have fled to less censorship-happy pastures.

It's also possible that young people are becoming more conservative overall. Conservatism is looking a lot better ever since the idiocy of the Bush administration got replaced by the idiocy of the Obama administration.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#26

Post by r721 » Mon, Sep 28, 2015, 9:26 AM UTC

grey-area wrote: post #20

tdkl wrote: post #14

Earlier quoted context omitted.

But we cannot download that one. This is actually what has been bothering me for a while now. In the past where search engines (well at least the big one) weren't that thorough, you didn't mind if something stayed on the Internet, since it wasn't easily accessible. But it's a major difference if it's indexed/searchable or downloadable.

But we cannot download that one. Not yet :) Given their lax security and general cluelessness of the people in charge, I'm quite sure it will leak at some point, at least partially, perhaps to corporations, perhaps to the public, just as internal NSA docs have leaked. It's very hard to keep things like that airtight forever - all it takes is one slip up and all the info stored could be accessible at some point in the…

I wonder whether they transformed top secret networks into intentionally slow networks now for large scale leaks to take years or tens of years :)

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#27

Post by tomtoise » Mon, Sep 28, 2015, 9:47 AM UTC

Houshalter wrote: post #11

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

This ties directly into the comment dataset, made by the same person and via the same method[1]. Working on pulling down the submission data now, will have to spin up another server if I want to do the comment/submission correlation.

>>"This dataset will go nicely with the full Reddit Comment Corpus that I released a couple months ago. The link_id from each comment corresponds to the id key in each of the submission objects in this dataset."

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#28

Post by rl3 » Mon, Sep 28, 2015, 10:13 AM UTC

lysium wrote: post #2

I'm not quite sure how I feel about it. Yes, that data was publicly available, but it 'feels' different that there is now an unalterable snapshot available to the public. No more comment deletion.

Agreed; this feels antithetical to Reddit's principles on some level.

When the next snapshot is released, it will be possible to just perform a diff to discover all deletions and edits since.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#29

Post by stared » Mon, Sep 28, 2015, 10:30 AM UTC

Is it it possible to directly use it with Spark? (I.e. without downloading it from AWS.) I mean, if the file is public, does it mean that there is a s3n:// address and it is accessible?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#30

Post by memeda » Mon, Sep 28, 2015, 10:47 AM UTC

️️

  /  l \
     /    l  \
   /      l    \
 /        l          \