Live data from Hacker News

Network visualization of 50k blogs and links

graph.henryn.ca

11–20 of 58 posts

Re: Network visualization of 50k blogs and links

#11
post #6
post #2

I scraped my favorite blogs and made a graph from the domains that each blog links to. You can see clusters forming of websites that talk about similar topics, like crypto, rationality, Canada, India, and even postgres! The visualization was made entirely in webgl with some neat optimizations to render that many lines and circles.

Very neat! So you wrote the graph visualization UI? I see in prior project you used cytoscape - any motivation for doing it yourself this time (vs one of the available libraries)?

Yeah I used cytoscape before but it didn't have the full customization that I wanted. Besides the performance issues, there were some problems I couldn't have solved without a custom renderer - if many lines overlap, how should their colors blend? - how to render circles so that they look nice both zoomed in / out - how to avoid it looking like a hairball graph [1]

The nice thing about a personal project is that I can do whatever I like with no constraints, so I built one that's suited for this project and fits my tastes.

[1] https://cambridge-intelligence.com/how-to-fix-hairballs/

Re: Network visualization of 50k blogs and links

#12
post #5

This is really awesome work! How did you classify so many links?

To get their topics? I used a basic louvain community detection algorithm, then put all the URLs into GPT with some few-shot prompting tricks to get it to output a particular topic. There's some heuristics to break up giant communities / combine small communities in there too.

Re: Network visualization of 50k blogs and links

#17
post #9
post #2

I scraped my favorite blogs and made a graph from the domains that each blog links to. You can see clusters forming of websites that talk about similar topics, like crypto, rationality, Canada, India, and even postgres! The visualization was made entirely in webgl with some neat optimizations to render that many lines and circles.

serendipity heaven, but... how is this map of _your favorite_ blogs?

I started off with my favorite blogs and recursively explored from there based on what they linked to.

Re: Network visualization of 50k blogs and links

#19

People still upvote hairball graphs every time. Fortunately there is a cure: https://cambridge-intelligence.com/how-to-fix-hairballs/

The hairball was much worse before. I used a lot of techniques from this paper [1] to make it look decent and a bunch of other heuristics based on other papers to make it look informative.

[1] https://jgaa.info/accepted/2015/NocajOrtmannBrandes2015.19.2...

Post reply on HN