Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

blog.wilsonl.in

Post reply on HN
Print view View original

71–80 of 171 posts

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#71

Post by minimaxir »

bravura wrote: post #66
minimaxir wrote: post #60

A modern recommendation for UMAP is Parametric UMAP ( https://umap-learn.readthedocs.io/en/latest/parametric_umap.... ), which instead trains a small Keras MLP to perform the dimensionality reduction down to 2D by minimizing the UMAP loss. The advantage is that this model is small and can be saved and reused to predict on unknown new data (a traditionally trained UMAP model is large), and training is theoetically muc…

From a quick glance, it appears that it's because the implementation pushes the entire graph (all edges) to the GPU. Sampling of edges during training could alleviate this.

Indeed, TensorFlow likes pushing everything to the GPU by default whereas many PyTorch DL implementations encourage feeding data from the CPU to the GPU as needed with a DataLoader.

There have been attempts at a PyTorch port of Parametric UMAP (https://github.com/lmcinnes/umap/issues/580) but nothing as good.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#72

Post by xanderlewis »

deadbabe wrote: post #65
wilsonzlin wrote: post #42

Earlier quoted context omitted.

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

Anecdotally, I think anyone who reads HN for a while will realize it to be a negative, cynical place. Posts written in sweet syrupy tones wouldn’t do well here, and jokes are in short supply or outright banned. Most people here also seem to be men. There’s always someone shooting you down. And after a while, you start to shoot back.

(Without wanting to sound negative or cynical) I don’t think it is, but maybe I haven’t been here long enough to notice. It skews towards technical and science and technology-minded people, which makes it automatically a bit ‘cynical’, but I feel like 95% of commenters are doing so at least in good faith. The same cannot be said of many comparable discussion forums or social media websites.

Jokes are also not banned; I see plenty on here. Low-effort ones and chains of unfunny wordplay or banter seem to be frowned upon though. And that makes it cleaner.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#73

Post by minimaxir »

Der_Einzige wrote: post #67
minimaxir wrote: post #60

A modern recommendation for UMAP is Parametric UMAP ( https://umap-learn.readthedocs.io/en/latest/parametric_umap.... ), which instead trains a small Keras MLP to perform the dimensionality reduction down to 2D by minimizing the UMAP loss. The advantage is that this model is small and can be saved and reused to predict on unknown new data (a traditionally trained UMAP model is large), and training is theoetically muc…

It exists in cuML with a fast GPU implementation. Not sure why cuMl is so poorly known though…

I'll give that a look: the feature set of GPU-accelerated ops seems just up my alley for this pipeline: https://github.com/rapidsai/cuml

EDIT: looking through the docs it's just GPU-acceletated UMAP, not a parametric UMAP which trains a NN model. That's easy to work around though by training a new NN model to predict the reduced dimensionality values and minimizing rMSE.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#74

Post by bravura »

minimaxir wrote: post #71
bravura wrote: post #66

Earlier quoted context omitted.

From a quick glance, it appears that it's because the implementation pushes the entire graph (all edges) to the GPU. Sampling of edges during training could alleviate this.

Indeed, TensorFlow likes pushing everything to the GPU by default whereas many PyTorch DL implementations encourage feeding data from the CPU to the GPU as needed with a DataLoader. There have been attempts at a PyTorch port of Parametric UMAP ( https://github.com/lmcinnes/umap/issues/580 ) but nothing as good.

Looks like there is a little motion on this topic:

https://github.com/lmcinnes/umap/pull/1103

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#75

Post by Lerc »

A suggestion for analysis:

Compare topics/sentiment etc. by number of users and by number of posts.

Are some topics dominated by a few prolific posters? Positively or negatively.

Also, How does one seperate negative/positive sentiment to criticism/advocacy?

How hard is it to detect positive criticism, or enthusiastic endorsement of an acknowledged bad thing?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#77

Post by sethammons »

xanderlewis wrote: post #72
deadbabe wrote: post #65

Earlier quoted context omitted.

Anecdotally, I think anyone who reads HN for a while will realize it to be a negative, cynical place. Posts written in sweet syrupy tones wouldn’t do well here, and jokes are in short supply or outright banned. Most people here also seem to be men. There’s always someone shooting you down. And after a while, you start to shoot back.

(Without wanting to sound negative or cynical) I don’t think it is, but maybe I haven’t been here long enough to notice. It skews towards technical and science and technology-minded people, which makes it automatically a bit ‘cynical’, but I feel like 95% of commenters are doing so at least in good faith. The same cannot be said of many comparable discussion forums or social media websites. Jokes are also not banned;…

I've been here a hot minute and I agree with you. Lots of good faith. Lots of personal anecdotes presumably anchored in experience. Some jokes are really funny, just not reddit-style. Similarly, no slashdot quips generally, such as "first post" or "i, for one, welcome our new HN sentiment mapping robot overlords." Sometimes things get downvoted that shouldn't, but most of the flags I see are well deserved, and I vouch for ones that I think are not flag-worthy

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#78

Post by Karrot_Kream »

beeboobaa3 wrote: post #68
wilsonzlin wrote: post #42

Earlier quoted context omitted.

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

[flagged]

Lol what a typical comment for today's HN. Condescending ("just plain wrong") with a jab ("this isn't a hugbox") placed in just to remind you that not only are you perceived to be wrong but you've provoked anger. No proof to provoke the jab, no feedback to help fix what you perceive as wrong sentiment analysis. Just thoughtless condescension and anger. Why is the sentiment wrong? Is this a data analysis trap the OP fell into? Nah let's insult the OP instead.

In my experience having run a bunch of different sentiment models on HN comments, HN comments tend to place around neutral to slightly negative as a whole, even when I perceive the thread to be okay. However I've noticed a huge bump in negative sentiment on large HN threads. I generally find that absolute sentiment doesn't work in most corpuses because the model reflects its training set's sentiment labels. I generally find relative sentiment to be a lot more useful. I have yet to do a temporal sentiment analysis on HN but I have a suspicion that it's gotten more negative over time. I agree with another poster that I think HN needs to be careful to not become so negative that it just becomes an anger echo.

Relative sentiment on this site between topics is something I've done and the obvious results show. Crypto threads are by-and-large negative, most political and news related threads are also highly negative.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#80

Post by the_sleaze_ »

Karrot_Kream wrote: post #78
beeboobaa3 wrote: post #68

Earlier quoted context omitted.

[flagged]

Lol what a typical comment for today's HN. Condescending ("just plain wrong") with a jab ("this isn't a hugbox") placed in just to remind you that not only are you perceived to be wrong but you've provoked anger. No proof to provoke the jab, no feedback to help fix what you perceive as wrong sentiment analysis. Just thoughtless condescension and anger. Why is the sentiment wrong? Is this a data analysis trap the OP f…

Cynicism is perceived as more intelligent [0]. I personally find the HN brand of discussion to be difficult to bs my way into. But no matter your level of competency you can always find something to criticize and feel you've contributed. I wonder if academia or even "more intelligent" discussion in general would be counted as more negative.

https://journals.sagepub.com/doi/pdf/10.1177/014616721878319...

Post reply on HN