Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

Very nice. Since Hn data spawns so many such fun projects, there should be a monthly or weekly updates zip file or torrent with this data, which hackers can just download instead of writing a scraper and starting from scratch all the time.

This is amazing, the amount of skill and knowledge involved is very impressive.

It was not obvious at first glance to me, but the actual app is here: https://hn.wilsonl.in/

Good example of data engineering/MLops for people who aren't familiar.

I'd suggest using HDBScan to generate hierarchical clusters for the points, then use a model to generate names for interior clusters. That'll make it easy to explore topics out to the leaves, as you can just pop up refinements based on the connectivity to the current node using the summary names.

The groups need more distinct coloring, which I think having clusters could help with. The individual article text size should depend on how important or relevant the article is, either in general or based on the current search. If you had more interior cluster summaries that'd also help cut down on some of the text clutter, as you could replace multiple posts with a group summary until more zoomed in.

Ok I just noticed there is a region "OpenAI" in the north-west, but for some reason it zooms in somewhere close to "Apple" (middle of the island) when I type the query