Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

blog.wilsonl.in

Post reply on HN
Print view View original

31–40 of 171 posts

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#33

Post by abe94 »

This is impressive work, especially for a one man show!

One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops?

I also found this sentence interesting, as it rings true to me about social media "there seems to be a lot of negative sentiment on HN in general." It would be cool to see a comparison of sentiment across social media platforms and across time!

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#34

Post by pfarrell »

thyrox wrote: post #2

Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.

I have a daily updated dataset that has the HN data split out by months. I've published it on my web page, but it’s served from my home server so I don’t want to link to it directly. Each month is about 30mb of compressed csv. I’ve wanted to torrent it, but don’t know how to get enough seeders since each month will produce a new torrent file (unless I’m mistaken). If you’re interested, send me a message. My email is mrpatfarrell. Use gmail for the domain.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#35

Post by wilsonzlin »

xnx wrote: post #24

As a novice, is there a benefit to using custom Node as the downloader? When I did my download of the 40 million Hacker News api items I used "curl --parallel". What I would like to figure out is the easiest way to go from the API straight into a parquet file.

I think your curl approach would work just as fine if not better. My instinct was to reach for Node.js out of familiarity, but curl is fast and, given the IDs are sequential, something like `parallel curl ::: $(seq 0 $max_id)` would be pretty simple and fast. I did end up needing more logic though so Node.js did ultimately come in handy.

As for the Arrow file, I'm not sure unfortunately. I imagine there are some difficulties because the format is columnar, so it probably wants a batch of rows (when writing) instead of one item at a time.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#36

Post by internetter »

noman-land wrote: post #21
graiz wrote: post #3

Would be cool to see member similarity. Finding like-minded commentors/posters may help discover content that would be of interest.

Accidental dating app.

> Accidental dating app.

Possibly the greatest indicator of social startup success.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#37

Post by pfarrell »

noman-land wrote: post #16
thyrox wrote: post #2

Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.

I very much support this idea. Put them on ipfs and/or torrents. Put them on HuggingFace.

I’ve had this same thought but was unsure what the licensing for the data would be.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#38

Post by wilsonzlin »

chossenger wrote: post #28

Awesome visualisation, and great write-up. On mobile (in portrait), a lot of longer titles get culled as their origin scrolls off, with half of it still off the other side of the screen - wonder if it'd be worth keeping on rendering them until the entire text field is off screen (especially since you've already got a bounding box for them). I stumbled upon [1] using it that reflects your comments on comment sentiment…

Thanks for the kind words, and raising that problem --- I've added it as an issue to fix.

Thanks for sharing that article, it was an interesting read. It was cool how deep the analysis went with a few simple statistical methods.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#40

Post by wilsonzlin »

ashu1461 wrote: post #17

This is pretty great. Feature request : Is it possible to show in the graph how famous the topic / sub topic / article is ? So that we can do an educated exploration in the graph around what was upvoted and what was not ?

Thanks! Do you mean within the sentiment/popularity analysis graph? Or the points and topics within the map?
Post reply on HN