Downloading All of Hacker News Posts and Comments • Hacker News Board

Re: Downloading All of Hacker News Posts and Comments

#31

Post by agibsonccc » Tue, Jun 03, 2014, 4:48 AM UTC

Thanks for the dataset! I've been wanting this corpus for a while. I have a few ideas I want to run with this.

Re: Downloading All of Hacker News Posts and Comments

#32

Post by [deleted] » Tue, Jun 03, 2014, 4:48 AM UTC

[deleted]

Re: Downloading All of Hacker News Posts and Comments

#33

Post by sytelus » Tue, Jun 03, 2014, 5:24 AM UTC

Just wanted to start a thread on some project ideas for this data:

* Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN.

* My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :).

* Find out abandonment rate of veteran users.

* Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in HN (for example, get links posted by people with 10K+ karma but without upvotes.

Re: Downloading All of Hacker News Posts and Comments

#34

Post by guernica » Tue, Jun 03, 2014, 5:35 AM UTC

sytelus wrote: post #33

Just wanted to start a thread on some project ideas for this data: * Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN. * My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). * Find out abandonment rate of veteran users. * Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in…

Construct phrases like Yoda, shall I. Unmask this Identity, never you shall.

Re: Downloading All of Hacker News Posts and Comments

#35

Post by frik » Tue, Jun 03, 2014, 5:44 AM UTC

simonwalton wrote: post #19

Great work! Does anyone know if there is anything similar for Slashdot?

The /. comments with its tags "Interesting", "Funny", "Insightful", "Informative", "Flamebait" can be very helpful for machine learning purposes.

Re: Downloading All of Hacker News Posts and Comments

#36

Post by sytelus » Tue, Jun 03, 2014, 6:06 AM UTC

meritt wrote: post #7

nekopa wrote: post #5

It's strange to see 1.3m stories and only 5.8m comments. I look forward to examining the data to see how many stories have 0 comments.

Lots and lots of stories never make it anywhere near the front page. Mostly spam.

Most stories are not spam, they are just posted at wrong time.

To give you an example, this story itself was posted on last Friday evening PST (https://news.ycombinator.com/item?id=7825146). It got just one upvote and 0 comments. Exact same story with exact same page content on exact same domain was posted on Monday (today) afternoon PST and it got 80+ upvotes, 30+ comments and got on frontpage for more than 6 hours!

A lot of stories are like this. HN ranking algorithm isn't perfect.

Re: Downloading All of Hacker News Posts and Comments

#37

Post by DanBC » Tue, Jun 03, 2014, 6:09 AM UTC

pmalynin wrote: post #20

SergeyHack wrote: post #15

Have you considered sharing the data as a torrent? The current FileDropper speed is quite low (~200KB/s) P.S. Thanks for the data, you saved a good amount of our time.

magnet:? xt=urn:btih:A3E2200A9A99906476E2E88CA002477219A3C2C3&dn=HN&tr= udp%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=udp% 3a%2f%2ftracker.publicbt.com%3a80%2fannounce&tr=udp%3a%2f%2ftr acker.ccc.de%3a80%2fannounce

PLEASE add four spaces before very long unbroken strings.

Re: Downloading All of Hacker News Posts and Comments

#38

Post by ciex » Tue, Jun 03, 2014, 9:56 AM UTC

snippyhollow wrote: post #16

ivansavz wrote: post #2

Awesome, this will be super useful for running ML experiments on the HN stories. Previously, Max Woolf worked on this http://minimaxir.com/2014/02/hacking-hacker-news/ via https://news.ycombinator.com/item?id=7291531

I also did topic modeling (LDA) and other experiments on HN, code for fetching all posts and comments, and for topic modeling, can be found here: https://github.com/SnippyHolloW/HN_stats (lein run // look at hn.py)

What were your results? What did you do it for?

Re: Downloading All of Hacker News Posts and Comments

#39

Post by jacquesm » Tue, Jun 03, 2014, 10:30 AM UTC

guernica wrote: post #34

sytelus wrote: post #33

Just wanted to start a thread on some project ideas for this data: * Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN. * My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). * Find out abandonment rate of veteran users. * Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in…

Construct phrases like Yoda, shall I. Unmask this Identity, never you shall.

Oh come on George, stop it.

Re: Downloading All of Hacker News Posts and Comments

#40

Post by jacquesm » Tue, Jun 03, 2014, 10:31 AM UTC

sytelus wrote: post #33

Just wanted to start a thread on some project ideas for this data: * Discover geek friendly WordPress themes and plugins by analyzing CSS in stories posted on HN. * My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :). * Find out abandonment rate of veteran users. * Find out undiscovered great stories that didn't got in to frontpage because algorithm deficiency in…

> My pet EVIL project: Extract self identifying statements from comments and create profile for HN users :).

I'd bet that you're not the only one who thought of that. And for less money I'd bet that someone has beat you to it.