Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Reddit Submission Corpus now available for 2006 thru August 2015

reddit.com

Post reply on HN
Print view View original

11–20 of 100 posts

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#11

Post by Houshalter »

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there.

Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone else has asked on reddit before. And the comment quality is usually pretty good. Sample conversation with it: https://i.imgur.com/LDD9isL.jpg

I improved on it a lot with a whitelist of subreddits and some machine learning to select the best thread. But I was only touching on what is possible with that data.

You can play with it here: https://kiwiirc.com/client/irc.snoonet.org/mybots

EDIT: It does only work well with certain kinds of questions. Thing's that would have been asked before and without too many unique keywords. I see some people ask it really unique questions, or talk to it like a regular chatbot without even asking questions. And then get frustrated when it returns nonsense. There are a lot of improvements that could be made with this with natural language processing and stuff. But right now it's pretty simple.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#12

Post by grey-area »

lysium wrote: post #2

I'm not quite sure how I feel about it. Yes, that data was publicly available, but it 'feels' different that there is now an unalterable snapshot available to the public. No more comment deletion.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too.

So comment deletion was never going to save you.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#13

Post by TorKlingberg »

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#14

Post by tdkl »

grey-area wrote: post #12
lysium wrote: post #2

I'm not quite sure how I feel about it. Yes, that data was publicly available, but it 'feels' different that there is now an unalterable snapshot available to the public. No more comment deletion.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

But we cannot download that one.

This is actually what has been bothering me for a while now. In the past where search engines (well at least the big one) weren't that thorough, you didn't mind if something stayed on the Internet, since it wasn't easily accessible.

But it's a major difference if it's indexed/searchable or downloadable.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#16

Post by creshal »

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

There sure as hell is lot more fear-mongering since the refugee "crisis" started.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#17

Post by CardenB »

ljk wrote: post #8
lysium wrote: post #2

I'm not quite sure how I feel about it. Yes, that data was publicly available, but it 'feels' different that there is now an unalterable snapshot available to the public. No more comment deletion.

> No more comment deletion Isn't that how hackernews is? not being able to delete comments

Yeah but I'm sure a lot of people post on here knowing that they won't delete a comment. It's a different story with Reddit, where people have a lot more expectations and it's bigger.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#18

Post by aunty_helen »

TorKlingberg wrote: post #13

I wonder if an analysis of this and the comment corpus could quantifiably show shifts in the opinions of the Reddit "hivemind" since the start. Has the Reddit userbase turned more conservative over the years, or is that just my impression?

It would also be interesting to see the development of astroturffing campaigns from a marketing and also a tinfoil hat perspective.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#19

Post by hippo8 »

tdkl wrote: post #14
grey-area wrote: post #12

Earlier quoted context omitted.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

But we cannot download that one. This is actually what has been bothering me for a while now. In the past where search engines (well at least the big one) weren't that thorough, you didn't mind if something stayed on the Internet, since it wasn't easily accessible. But it's a major difference if it's indexed/searchable or downloadable.

This is why, I think we need to make ICT (Information Communication Technology) a mandatory subject in our schools.

Teach kids about the internet. Many people are still under the assumption just deleting something would make it disappear for ever.

Maybe a lot of people from our generation are doomed to sharing too many things online, but we can at least save the next generation from themselves.

You make a mistake 10 years ago, may be a few close friends in your town know about you. Now you make a mistake, the whole world has access to that information.

Government regulations, bans are not going to do anything to stop the spread of information, we need to educate people to protect themselves from their own selves.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#20

Post by grey-area »

tdkl wrote: post #14
grey-area wrote: post #12

Earlier quoted context omitted.

There's also a backup copy in Utah with all your comments from every site tied to common selectors like your usernames, emails, passwords, ip, credit cards etc. No doubt that'll leak at some point too. So comment deletion was never going to save you.

But we cannot download that one. This is actually what has been bothering me for a while now. In the past where search engines (well at least the big one) weren't that thorough, you didn't mind if something stayed on the Internet, since it wasn't easily accessible. But it's a major difference if it's indexed/searchable or downloadable.

But we cannot download that one.

Not yet :)

Given their lax security and general cluelessness of the people in charge, I'm quite sure it will leak at some point, at least partially, perhaps to corporations, perhaps to the public, just as internal NSA docs have leaked. It's very hard to keep things like that airtight forever - all it takes is one slip up and all the info stored could be accessible at some point in the future. It's already indexed and searchable, just not by you.

The important point here is that looking at present day tech (as in your comment on search engine prowess) is not the way to look at it - one day all this information will be accessible to much more intelligent future algorithms, able to link it together in myriad ways and form an almost perfect picture of your life in retrospect. The data is there, and will be stored forever.

Post reply on HN