Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

blog.wilsonl.in

Post reply on HN
Print view View original

121–130 of 171 posts

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#121

Post by wilsonzlin »

dylan604 wrote: post #93
wilsonzlin wrote: post #42

Earlier quoted context omitted.

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

The sentiment issue is a curious one to me. For example, a lot of humans I interact with that are not devs take my direct questioning or critical responses to be "negative" when there is no negative intent at all. Pointing out something doesn't work or anything that the dev community encounters on a daily basis isn't an immediate negative sentiment but just pointing out the issues. Is it a meme-like helicopter parent…

You may be right, a more tailored classifier for HN comments specifically may be more accurate. It'd be interesting to consider the classes: would it still be simply positive/negative? Perhaps constructive/unconstructive? Usefulness? Something more along the lines of HN guidelines?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#122

Post by Swizec »

gieksosz wrote: post #119
abe94 wrote: post #33

This is impressive work, especially for a one man show! One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops? I also found this sentence interesting, as it rings true to me about social media "there seems to be a…

HN is a pretty toxic place indeed.

> HN is a pretty toxic place indeed

This may be a personal style difference, but I find HN to be the least toxic of all social media I’ve tried. LinkedIn would be my example of ultra toxicity – the aggressive positivity there is unbearable. At least on HN people tell you what they think and even use a constructive decently argumented approach to doing so.

HN to me feels like a good technical discussion where people tear apart ideas instead of each other.

But yeah if you put a lot of ego into your ideas, HN must be an awful place to visit.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#123

Post by remram »

thyrox wrote: post #2

Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.

As a starting point, that project has Apache Arrow files. I don't know if they'll update them though.

https://github.com/wilsonzlin/hackerverse/releases/tag/datas...

The comments text table is 13 GB, to give you an idea. Can definitely be processed on a laptop.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#124

Post by swatcoder »

gieksosz wrote: post #119
abe94 wrote: post #33

This is impressive work, especially for a one man show! One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops? I also found this sentence interesting, as it rings true to me about social media "there seems to be a…

HN is a pretty toxic place indeed.

How did you get from negative sentiment to toxicity? Are those the same to you?

It may be a cultural thing, but I think many people see negative sentiment as a constructive tool and a demonstration of trust and respect among people who recognize each others as robust and capable peers.

Avoiding it is something you do with people who you believe need special delicacy: whether because they've told you so, because they intimidate you, or because you sense something pitiable and fragile about them.

If you can trust that it's given in good faith, and by the guidelines of HN you are asked to, negative sentiment should be seen as an expression that someone thinks you're a fully capable adult and peer. Personally, I deeply appreciate that it's generally so comfortably shared and received here and would never include "toxicity" in one of my critiques of HN.

It's a surprising thing to read someone say!

(Unless you're thinking of the nastiness that can surface on flamewar topics, but there are numerous means by which those get downranked and displaced, and they're otherwise sparse and easy to avoid.)

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#125

Post by ComputerGuru »

Amazing work, I'm impressed by the scope of your project!

I must say though, is it jina or bge-3/flag - the embeddings (and tokenizer?) do not do a good job on tech topics. It's fine for natural words, but searching for tech concepts like "xaml", "simd", etc cause it fall back to tokenizing the inputs and tries to grab similar sounding words.

Also, just some constructive feedback, if there were some way to stop it from showing the same "hn leaderboard" of results when there are no results because a topic is too niche would be nice. I get a lot of "Stephen Hawking has died" when searching for words the embeddings aren't familiar with.

Edit: I'm not so sure how well the sentiment analysis is working. I had the feeling that there was too much negative sentiment that didn't match up to reality, so I tried looking up things HN would feel overwhelmingly positive about like "Mr Rogers", I mean, who could feel negatively about him? The results show some serious negative spikes. Look up "Carter" and there's a massive negative peak associated with the passing of Rosalynn Carter. It was an HN submission talking about all the wonderful things the Carters did.

Also, I think the "popularity over time" needs to be scaled by the median number of votes a story got that month/year, because the trend lines just go up and up if you plot strictly the number of posts. Look at the popularity of "diesel" and you'll see what I mean - this is a term that peaked ten years ago! Or perhaps it should be some sort of keyword incidence rate or number of items with a cosine similarity index of less than x from the query rather than post score, maybe?

Edit2: The dynamic "click a post to remove and recalculate similarity threshold" is awesome.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#128

Post by al_hag »

wilsonzlin wrote: post #42
abe94 wrote: post #33

This is impressive work, especially for a one man show! One thing that stood out to me was the graph of the sentiment analysis over time, I hadn't seen something like that before and it was interesting to see it for Rust. What were the most positive topics over time? And were there topics that saw very sudden drops? I also found this sentence interesting, as it rings true to me about social media "there seems to be a…

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

It will be a deep dive into the most essential of HN staples, the nitpick

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#129

Post by rantymcrant »

I'd like to see an analysis of the rise of self promotion on HN.

I define self promotion on HN as a "Show HN: I ..." post vs "Show HN: Something ..."

Examples from the top 100 right now

* "Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun"

* "Show HN: Browser-based knitting (pattern) software"

These are not self promotional titles. The subjects are the exploration and the software respectively.

* "Show HN: I built a non-linear UI for ChatGPT"

* "Show HN: I created 3,800+ Open Source React Icons"

These are self promotional titles. The subject of each is "I"

My own simple check just via algolia search results checking for titles that start with "Show HN: I" gave these results for years starting April 1st. Graphed divided by the total number of results for that year

    2023 ****************************************
    2022 ***********************************
    2021 ***************************
    2020 **************************************
    2019 *************************
    2018 *************
    2017 *******
    2016 **********
    2015 ********
    2014 ************
    2013 *********************
    2012 *****************
    2011 *********
    2010 ***
I feel like maybe I grew up in a time when generally, self promotion was considered a bad character trait. Your actions are supposed to be what promotes you, calling attention to them is not but I feel that culture is changing.

I wonder if the rise in self promotion (assuming there is a rise) has to do with social media etc...

I perceive a similar rise on Youtube but I have no data, just a feeling from the number of youtube recommendations for videos of "I....."

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#130

Post by Thorrez »

rantymcrant wrote: post #129

I'd like to see an analysis of the rise of self promotion on HN. I define self promotion on HN as a "Show HN: I ..." post vs "Show HN: Something ..." Examples from the top 100 right now * "Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun" * "Show HN: Browser-based knitting (pattern) software" These are not self promotional titles. The subjects are the exploration and the software respecti…

Your definition of self promotion is a bit different from what I usually think. I usually consider self promotion to be someone promoting something that that same person did. Both of your non-self-promotion examples would be self promotion under my definition.

So what you consider to be self promotion vs non-self-promotion, I consider to be self promotion with a title that very clearly indicates that vs self promotion with a title that less clearly indicates that. However, the "Show HN" phrase is only used for self promotion I think, so even without the "I", anyone familiar with the convention will know it's self promotion.

Post reply on HN