Show HN: Analyzing top HN posts with language models

Hi HN,

I spent a few weeks looking at the top HN posts of all time. This included exploration, clustering, creating visualizations, and zooming in on what (to me personally) seems like some of the best discussions on here.

Three things in this post:

1- The interesting groups of HN posts

2- The interactive visualizations that you can explore in your browser

3- The data from this exploration -- this includes CSV of the titles as well as the text embeddings of 3,000 Ask HN articles.

Blog post about this whole process here: [1]

============

1- The interesting groups of HN posts

From the exploration, Ask HN proved the most interesting. These are the top four groups of topics I found insightful. Each group contains about 400 posts.

- Life experiences and advice threads [2]

- Technical and personal development [3]

- Software career insights, advice, and discussions [4]

- General content recommendations (blogs/podcasts) [5]

============

2- The interactive visualizations that you can explore in your browser

- Top 10,000 Hacker News articles of all time [6]

- Top 3,000 posts in Ask HN [7]

============

3- The data from this exploration

CSV file of top 3K Ask HN posts: [8]

The sentence embeddings of the titles of those posts: [9]

This is a colab notebook containing the code examples (including loading these two data files): [10]

============

If you've ever wanted to get into language models, this is a good place to start. Happy to answer any questions

I don't know how HN score metrics work but after some short review of the datafile [1] I've noticed a lot of the posts has the form of a simple questions and as such seems to be naturally biased when comes to user engagement. Have you considered to add additional metrics to remove that bias and re-analyze?

[1] https://storage.googleapis.com/cohere-assets/blog/text-clust...

So it really depends on what you use for clustering. In this case, I'm clustering by the original embeddings so the UMAP results are different. I've also seen:

1- Clustering by UMAP. Here the plot would show clean separation of topics. But the clustering algorithm would be working on highly compressed data (from the 1024 dimensions of the embedding down to the 2 of UMAP).

2- BERTopic's approach of doing UMAP down to 5 dimensions, using this dimensionality for clustering, then UMAP again from 5 to 2. Which is an interesting approach.

I've heard people having good results with all three. It's kinda hard to objectively compare, but my leaning was to give the clustering algorithm the representation containing the most information about the text.

Right, bertopic's double clustering is interesting. I've also seen people combine that with louvain instead of k-means.

My intuition was: UMAP itself tries to optimize for 2d separation in the projection. So we should expect at least some correspondence between the kmeans results and the layout in the UMAP plot (except in some pathological edge cases perhaps).

Nevertheless, nice example and blog post!

Are you sure t-SNE and UMAP actually perform very differently? Last I looked, they were somewhat comparable.

[edit]: Seems they are similar for some purposes: https://blog.bioturing.com/2022/01/14/umap-vs-t-sne-single-c...

Also interesting: Rapidsai has a cuda accelerated version of umap that is very fast (hdbscan as well BTW).