From the article: "The web definitely has a speed problem due to over-design and the junkyard of tools people feel they have to include on every single web page. However, I don’t agree that the web has an inherent slowness. The articles for the new Facebook feature will be sent over exactly the same connection as web pages. However, the web versions of the articles have an extra layer of cruft attached to them, and t…
I saved a Washington Post page -- a short interview -- and it came to 3054KB. I saved a NY Post page and it came to 2843K. For all I know, there could have been piles of background junk that weren't saved with the page. It does seem a bit remarkable that these sites need to ship 3 megabytes of stuff to deliver 20K of text.... I think I need your web service ;-)
http://www.sitetruth.com/fcgi/viewer.fcgi?url=news.ycombinator.com
replacing the URL parameter with the desired web site. It's a tool we use to see how our crawler views a page. It reads the first 1MB from a URL, converts everything to UTF-8 based on the strict rules for charsets (no guessing), parses it with a HTML5 parser to create a parse tree, throws out the Javascript and Flash, makes all the links absolute, and emits a pretty-printed version.Most sites have readable text after this, although some features may not work. The ones that don't do something like creating the page with document.write(), or one of those "parallax" formats. It's a reasonable test to see if the web designers went completely overboard. If you're getting a blank page, search engines probably aren't indexing that page very well.