Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full Reddit Submission Corpus now available for 2006 thru August 2015

reddit.com

Post reply on HN
Print view View original

41–50 of 100 posts

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#41

Post by voltagex_ »

tomtoise wrote: post #40

I've spent some time turning this dataset into a Torrent - magnet link is here. If the Reddit OP doesn't want it to be shared in this way for whatever reason, I will of course kill it. Just trying to save him some bandwidth. magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&ws=http://reddit-data.s3.amazonaws.com/RS_full_corpus.bz2 adds the S3 link as a web seed.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#42

Post by tomtoise »

voltagex_ wrote: post #41
tomtoise wrote: post #40

I've spent some time turning this dataset into a Torrent - magnet link is here. If the Reddit OP doesn't want it to be shared in this way for whatever reason, I will of course kill it. Just trying to save him some bandwidth. magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80

magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&ws= http://reddit-data.s3.amazonaws.com/RS_full_corpus.bz2 adds the S3 link as a web seed.

That's pretty cool. So would it pull the data from the web or the seedbox? Sorry - I'm fairly newbish with p2p technology.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#43

Post by voltagex_ »

tomtoise wrote: post #42
voltagex_ wrote: post #41

Earlier quoted context omitted.

magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&ws= http://reddit-data.s3.amazonaws.com/RS_full_corpus.bz2 adds the S3 link as a web seed.

That's pretty cool. So would it pull the data from the web or the seedbox? Sorry - I'm fairly newbish with p2p technology.

Both, if the client is new enough. I'm not sure what weight is given to each. You have no contact details and the site on your keybase profile is dead - can you get in touch? I'd like to experiment with this.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#44

Post by tomtoise »

tomtoise wrote: post #42
voltagex_ wrote: post #41

Earlier quoted context omitted.

magnet:?xt=urn:btih:9941b4485203c7838c3e688189dc069b7af59f2e&dn=RS%5Ffull%5Fcorpus.bz2&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&ws= http://reddit-data.s3.amazonaws.com/RS_full_corpus.bz2 adds the S3 link as a web seed.

That's pretty cool. So would it pull the data from the web or the seedbox? Sorry - I'm fairly newbish with p2p technology.

Shooting you an email to the email listed on your profile. I really should fix my keybase.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#46

Post by laumars »

tomtoise wrote: post #38
laumars wrote: post #36

Earlier quoted context omitted.

Is the Comment Corpus still available? I have looked but couldn't see anything obvious (my apologies if I'm being thick here)

Sure, here you go [1] - It unpacks to over 1tb though, hence me considering spinning up another server just for the comments. Might be a good use for my test Hadoop cluster. [1] https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_eve...

Thank you. I'll be unpacking this on a FreeBSD server, so hopefully ZFS's lz4 file system compression will take some of the bite out of that 1TB file.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#47

Post by [deleted] »

Houshalter wrote: post #11

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

[deleted]

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#49

Post by stuck_in_the_ma »

marcosscriven wrote: post #15

I couldn't see from the source what form the data are in? It mentions what properties are available, but is it a DB snapshot? Are 'submission objects' only text?

Hi there. I'm the one that released this. The data is just a bunch of JSON objects separated by new lines (\n). It's basically the exact same info you would get in JSON format from Reddit's API.

Re: Full Reddit Submission Corpus now available for 2006 thru August 2015

#50

Post by stuck_in_the_ma »

Houshalter wrote: post #11

This is not as cool as the comment dataset, but it's still pretty cool. I think in the future AIs will make extensive use of these kinds of datasets. It's amazing how much useful information is in there. Last year I made a simple IRC question answering bot that just searched reddit for your question and returned the top comment of the top result. It worked surprisingly well. It's amazing how many questions someone el…

This is very cool! Glad to see that people are making good use of this. Are you pulling the answer with the highest score?
Post reply on HN