Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A dataset of every Reddit comment

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 96 posts

Re: A dataset of every Reddit comment

#24

Post by curiousjorge »

minimaxir wrote: post #23
curiousjorge wrote: post #21

how to access the BigQuery data? I get this screen Welcome to BigQuery! What is BigQuery? but it doesn't show the data.

You'll need to set up a project in BigQuery first. (you don't have to give billing information to use the free quota)

got it running!

I just did a query for my reddit handle and it took 6.5 seconds to retrieve all of my comments. Kind of a snowden moment but this is super interesting, first time I played with Big Query.

Would love to run some google API for sentiment analysis.

Re: A dataset of every Reddit comment

#25

Post by thomasfromcdnjs »

avinassh wrote: post #22
thomasfromcdnjs wrote: post #15

Someone should try make a sentient twitter bot that learns from the Reddit data

How does one go about doing that? I know how to program, but I have written anything which do 'intelligent' stuff.

I have no idea either, I would also love for someone to chime in with some first steps.

Re: A dataset of every Reddit comment

#26

Post by Ianvdl »

thomasfromcdnjs wrote: post #25
avinassh wrote: post #22

Earlier quoted context omitted.

How does one go about doing that? I know how to program, but I have written anything which do 'intelligent' stuff.

I have no idea either, I would also love for someone to chime in with some first steps.

Markov chains would probably be a good start.

See reddit.com/r/subredditsimulator (not always safe for work)

Re: A dataset of every Reddit comment

#28

Post by daave »

minimaxir wrote: post #16

As linked below, I've played around a bit with this dataset: https://www.reddit.com/r/dataisbeautiful/comments/3cjyvb/rel... BigQuery is the best interface for it. Can resolve queries on the entire dataset in less than a few seconds (however, you only get 1TB processing free per month. Since the full dataset is ~285GB, you only get 4 queries per month. Plan ahead on the May 2015 dataset, which is only 8GB.) I can ans…

> Since the full dataset is ~285GB, you only get 4 queries per month.

That's only true if your 4 queries need to read every single column.

One of the big advantages of BigQuery's column-oriented storage is that you only pay to read the columns that are actually needed to answer your query.

For example, this query to extract the top 10 authors only cost me 19GB to run (and took 7.0s):

  SELECT
    author,
    COUNT(*) AS COUNT
  FROM
    TABLE_QUERY([fh-bigquery:reddit_comments], "table_id CONTAINS '20' AND LENGTH(table_id)

Re: A dataset of every Reddit comment

#29

Post by TTPrograms »

This would be a really amazing way to make factually backed statements about the nature of Reddit for news sources given the recent publicity.

I.e. the frequency of comments of a certain nature, typical karma scores for those comments, breakdown by subreddit etc.

Post reply on HN