Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A dataset of every Reddit comment

news.ycombinator.com

Post reply on HN
Print view View original

91–96 of 96 posts

Re: A dataset of every Reddit comment

#91

Post by zo1 »

kristopolous wrote: post #78
nathan_f77 wrote: post #18

I think it's fascinating that if anyone uses this dataset to train an AI, some tiny piece of my personality might make a contribution.

It's going in my /raid/datasets with about 1.4TB of other comment data. So the answer is going to be "yes" ... well probably by Tuesday morning it will be. I've used older reddit dumps before ... it's not great data for most things, but there's an awful lot of it!

What are you going to use it for? If you don't mind letting us in on some details, however vague albeit interesting.

Re: A dataset of every Reddit comment

#92

Post by ytdht »

Houshalter wrote: post #90
ytdht wrote: post #88

Earlier quoted context omitted.

here it is, only selected the body of your comments though, out of 4005 comments... http://s000.tinyupload.com/?file_id=09238386475544637092 here is the query that I used: SELECT author, body FROM [fh-bigquery:reddit_comments.2007], [fh-bigquery:reddit_comments.2008], [fh-bigquery:reddit_comments.2009], [fh-bigquery:reddit_comments.2010], [fh-bigquery:reddit_comments.2011], [fh-bigquery:reddit_comments.2012], [fh-big…

Thank you so much! I'd like to try it myself, but your link redirects me to some getting started page. Any idea why some comments are missing? I haven't checked to see if they are all there, as far as I know they are. I know that any in closed subreddits or comments that were removed by mods might be missing. But if you can access the comment's permalink without being signed in, then it should be in your data.

No problem. I believe that you need to create a project first, for the page to work.

You were right and that comment has been deleted for some reason.

Re: A dataset of every Reddit comment

#93

Post by kristopolous »

zo1 wrote: post #91
kristopolous wrote: post #78

Earlier quoted context omitted.

It's going in my /raid/datasets with about 1.4TB of other comment data. So the answer is going to be "yes" ... well probably by Tuesday morning it will be. I've used older reddit dumps before ... it's not great data for most things, but there's an awful lot of it!

What are you going to use it for? If you don't mind letting us in on some details, however vague albeit interesting.

sure. http://frustrometer.com

Re: A dataset of every Reddit comment

#94

Post by ers35 »

ers35 wrote: post #73

I wanted the data in a form I could easily query, so I wrote a program to convert the JSON to an SQLite database: https://gist.github.com/ers35/3b615a75fa0ed5e6d5cc I have the program running on Amazon EC2 right now converting the whole dataset. I plan to upload the database to the Internet Archive when it completes.

Here it is: https://archive.org/details/2015_reddit_comments_corpus_sqli...
Post reply on HN