Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Network visualization of 50k blogs and links

graph.henryn.ca

Post reply on HN
Print view View original

51–58 of 58 posts

Re: Network visualization of 50k blogs and links

#51

Post by mikk14 »

ng-henry wrote: post #19
PaulHoule wrote: post #14

People still upvote hairball graphs every time. Fortunately there is a cure: https://cambridge-intelligence.com/how-to-fix-hairballs/

The hairball was much worse before. I used a lot of techniques from this paper [1] to make it look decent and a bunch of other heuristics based on other papers to make it look informative. [1] https://jgaa.info/accepted/2015/NocajOrtmannBrandes2015.19.2...

I'd give it a shot to make node embeddings with Node2Vec [1] and then reduce them to 2D with UMAP [2]. I think it could help breaking apart the hairball, assuming you have a nice clustered structure.

[1] https://pytorch-geometric.readthedocs.io/en/latest/generated... [2] https://umap-learn.readthedocs.io/en/latest/

Re: Network visualization of 50k blogs and links

#52

Post by dougunplugged »

internetter wrote: post #24
ng-henry wrote: post #12

Earlier quoted context omitted.

To get their topics? I used a basic louvain community detection algorithm, then put all the URLs into GPT with some few-shot prompting tricks to get it to output a particular topic. There's some heuristics to break up giant communities / combine small communities in there too.

Interesting, I was curious what I would be categorized as and it's "Whistleblowing and Leaks", which I do suppose is what my content has lately been to some extent but it was funny to see that written out. My question for you is how can I see what sites link to me, as opposed to what sites I link to?

Sites like ahrefs.com, semrush.com and majestic.com can show what sites link to your site. Looks like your site boehs.org is pretty popular with thousands of backlinks. Your top linked pages are: https://boehs.org/node/everything-i-know-about-the-xz-backdo... https://boehs.org/node/truth-social https://boehs.org/node/npm-everything

Re: Network visualization of 50k blogs and links

#53

Post by pmayrgundter »

jmmv wrote: post #45

Reminds me of the intern project I worked on at Google back in 2008. My mentor at the time had a traceroute dataset of the Internet and wanted to render it on top of Google Maps. I implemented a MapReduce algorithm that geolocated the data points and then produced Google Maps tiles at various zoom levels to show how the Internet was connected. It was pretty cool to visualize how the data flowed throughout the world a…

Howdy sir! Great memory and fun times :)

Dataset was something from CAIDA, like this: https://www.caida.org/catalog/datasets/ipv4_prefix_probing_d...

IIRC we used the LGL algorithm (https://lgl.sourceforge.net/) while pinning any nodes we could get geolocations for, giving a nice hybrid geo/topo layout

I don't remember exactly how we got the geolocations, but often network routers have 3-letter airport codes in their DNS names, so maybe that? We may also have had a lookup table in el googz somewhere

Definitely a project whose time should again come! ;)

Re: Network visualization of 50k blogs and links

#54

Post by jmmv »

pmayrgundter wrote: post #53
jmmv wrote: post #45

Reminds me of the intern project I worked on at Google back in 2008. My mentor at the time had a traceroute dataset of the Internet and wanted to render it on top of Google Maps. I implemented a MapReduce algorithm that geolocated the data points and then produced Google Maps tiles at various zoom levels to show how the Internet was connected. It was pretty cool to visualize how the data flowed throughout the world a…

Howdy sir! Great memory and fun times :) Dataset was something from CAIDA, like this: https://www.caida.org/catalog/datasets/ipv4_prefix_probing_d... IIRC we used the LGL algorithm ( https://lgl.sourceforge.net/ ) while pinning any nodes we could get geolocations for, giving a nice hybrid geo/topo layout I don't remember exactly how we got the geolocations, but often network routers have 3-letter airport codes in the…

Oh that’s right! We had to use LGL in addition to geolocation to lay out points without data. And yeah, the geolocation came from some internal service.

Thanks for chiming in. Good times!

Re: Network visualization of 50k blogs and links

#55

Post by abalaji »

ng-henry wrote: post #7
abalaji wrote: post #4

This is neat, found my blog in there. Don't think I linked to NatGeo at any point, though. adithyabalaji.com

Nice! I looked through the logs and saw that you linked to it in this article: https://www.adithyabalaji.com/datascience/2021/05/17/Analyzi...

ah, you’re right!

Re: Network visualization of 50k blogs and links

#57

Post by ploum »

ng-henry wrote: post #32
jszymborski wrote: post #20

My blog is on here, but as a lonely, lonely node. I link to stuff I promise!

We just haven't crawled your site yet! There's a lot of links so we can't crawl them all :(

Well, the results are quite strange. Mine ( ploum.net ) is said to have 7 links to youdoblog. (which is false, there’s only one link to that website in all my 900 blog posts).
Post reply on HN