Some analysis of the 1M most popular sites on the web

Post by lifeisstillgood » Fri, Jul 24, 2015, 7:58 PM UTC

mixologic wrote: post #61

jacquesm wrote: post #37

Earlier quoted context omitted.

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

It is pretty standard practice to host assets on a "cookieless" domain you control, but not on the same domain as the original site. For example, www.example.com has all the html, but all of the images are hosted at www.images-example.com. That would skew the results considerably.

Post by [deleted] » Fri, Jul 24, 2015, 8:02 PM UTC

Paul-ish wrote: post #97

jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

>This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely above 10. So we are hoping that the user will have a cached copy of that exact jQuery version from that exact CDN. I wonder if it might be a go…

Post by lifeisstillgood » Fri, Jul 24, 2015, 8:09 PM UTC

jacquesm wrote: post #111

lifeisstillgood wrote: post #110

Earlier quoted context omitted.

Thanks! Two things leap out. Firstly I love the way you chose to do 1 million sites. I would have gone, hmm, maybe top thousand, and called it a representative sample :-) The scale of the modern world is still something I am grappling with. Secondly, is that 200 Mbps down / 20 mbps up? I think the UK has some broadband access lessons to learn if that's true. My wet piece of string is getting threadbare.

It's maybe overkill to do it on the whole set instead of just a sample, probably the numbers would not change all that much. The 200/20 is indeed 200 Mbps down and 20 up, this little trick saturated the line pretty good though. I probably could have saved some time and bandwidth by letting phantomjs abort on image content but I was lazy.

Post by jfb » Fri, Jul 24, 2015, 8:44 PM UTC

andrewljohnson wrote: post #66

jacquesm wrote: post #57

Earlier quoted context omitted.

Explain to me like I'm five what features a website that hosts it's own javascript can't have versus one that loads those same javascripts from remote source?

It can't have the features that would have been built, in the time spent learning about and implementing security. I regard nearly all security for startup-class, low-user, and low-value companies to be premature optimization, which is deadly to a new project's potential.

Post by [deleted] » Fri, Jul 24, 2015, 9:59 PM UTC

jacquesm wrote: post #76

johncolanduoni wrote: post #70

Earlier quoted context omitted.

I understand that it isn't possible to check if the external assets are hosted on a CDN bucket which is under the control of the website (but under a different domain name), but without the ability to discriminate such cases it makes your statistics on externally hosted content pretty meaningless.

I don't agree with that. From the point of view of the user that content might as well have come from a third party since - just like the headless browser used in the testing - they have absolutely no way to verify that short of doing a bunch of whois lookups. And if there is one thing that a user should be able to verify then it is that the entity sending them the main page is the same entity as the one that sends t…