11–20 of 53 posts
Apparently it uses Google Maps for display, and I wonder, does GMaps API allow use of custom map data, or does it mean that this visualization is made by Google?
> The Internet map is a non-commercial project. You can share our expenses and let more people see beauty of the Internet.
I don't think it's a Google project.
Apparently it uses Google Maps for display, and I wonder, does GMaps API allow use of custom map data, or does it mean that this visualization is made by Google?
This is the Internet map a few months after I first got on from Harv-10 in 1972: http://www.flickr.com/photos/walkingsf/7257339850/lightbox/ . Brings back great memories of sitting by the Harvard IMP (router) late at night, and getting a call from BBN on the phone asking me (anyone) to reboot it manually...
I'd love to see a how this was made blog article... especially how all of the data was found/processed.
Basically, they took a web crawler like Heritrix (archive.org) or Scrapy (a handy Python implementation good for prototyping) and just started fetching web pages.
Eventually, they have a database of 350,000 websites, along with two million links between these domains. Any set of web pages within a given domain may have hundreds of hyperlinks to a dozen other domains, but a link from any page in one domain to any page in another domain becomes a relationship between two domain nodes in a graph. Presumably they used something like neo4j.org to store these relationships (cf. jokes about relational databases being bad at storing relationship information).
Then the actual hard part comes in. They link to a high level paper on rendering a visualization of that much information, and then used a similar algorithm to determine placement of each node. The size of each node is presumably the number of links in/out and the color coding is geographic (and probably not considered in this algorithm).
So now they have a database describing all these nodes and relationships, and an algorithm to draw a gigantic image of all of them in 2D space. They used GPU-based parallel processing techniques (probably with NVidia's CUDA language) to crunch all the numbers to generate the final image.
Finally, the image ends up being pretty large at a reasonable zoom. A scaled map of the Earth zoomed a bit above street level would still be about 125 miles on a side. So they use Google Maps API to manage small chunks of the image at various zoom levels. (They also end up rerunning that algorithm a few times to generate smaller images for each zoom step, including one at good old 1024x768).
Pretty neat. Would love to see their writeup.
Apparently it uses Google Maps for display, and I wonder, does GMaps API allow use of custom map data, or does it mean that this visualization is made by Google?
This map uses a google.maps.ImageMapType [2] along with a custom EuclideanProjection map projection [3] which replaces the standard spherical Mercator ("Google Mercator") projection.
[1] https://developers.google.com/maps/documentation/javascript/...
[2] https://developers.google.com/maps/documentation/javascript/...
[3] View source of http://internet-map.net/
I'd love to see a how this was made blog article... especially how all of the data was found/processed.
The about page has a fair amount of detail, but a rough outline: Basically, they took a web crawler like Heritrix (archive.org) or Scrapy (a handy Python implementation good for prototyping) and just started fetching web pages. Eventually, they have a database of 350,000 websites, along with two million links between these domains. Any set of web pages within a given domain may have hundreds of hyperlinks to a dozen…
What is it visualizing? Pages or links? Because if it's links, there's no way that Facebook is so large and Wikipedia is so small.
This is actually a very useful advertising tool - if you're considering placing ads on any sites, it's a great way to see how they compare to other sites in terms of traffic and visitor flow.