Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some analysis of the 1M most popular sites on the web

jacquesmattheij.com

Post reply on HN
Print view View original

61–70 of 130 posts

Re: Some analysis of the 1M most popular sites on the web

#61

Post by mixologic »

jacquesm wrote: post #37
gingerlime wrote: post #21

I wonder what's considered external though. If I compile / minify my javascript and CSS and then use a CDN to cache or host it -- is this considered external?? If so, how is it different from trusting my hosting provider to host my site in the first place, or my domain provider for resolving for me? How can this analysis know whether or not the resource is external? based on dns records alone? because I can still use…

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

It is pretty standard practice to host assets on a "cookieless" domain you control, but not on the same domain as the original site. For example, www.example.com has all the html, but all of the images are hosted at www.images-example.com. That would skew the results considerably.

Re: Some analysis of the 1M most popular sites on the web

#62

Post by corford »

jacquesm wrote: post #37
gingerlime wrote: post #21

I wonder what's considered external though. If I compile / minify my javascript and CSS and then use a CDN to cache or host it -- is this considered external?? If so, how is it different from trusting my hosting provider to host my site in the first place, or my domain provider for resolving for me? How can this analysis know whether or not the resource is external? based on dns records alone? because I can still use…

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

What about the common trick of serving static assets via a separate domain (one that's still owned and hosted by you) to avoid the overhead of cookies from the main site being included in asset requests?

Re: Some analysis of the 1M most popular sites on the web

#63

Post by captainmuon »

One thing in this context is that it is basically impossible for a website to check the integrity of an external (js) resource without loading it. This is a consequence of the web security model.

Its basically impossible to get the contents of a .js file without executing it, say for checksum verification (at least without CORS, and even with, you might trigger an additional download, I haven't tested it). But it's trivially easy to include an external .js in the page, with the same access rights as directly embedded script (including access to credentials).

That's what we're used to, but it seems completely backwards to me. I would be much better IMO if a script could make arbitrary HTTP requests to other sites - but without having access to those sites' credentials. (Remember in the 2000s when "mashups" were all the rage? I spent a weekend parsing some data source in javascript to display it on a map, just to realize that what worked locally didn't work over http. Imagine the disappointment.)

What's also missing is a way to run an external script sandboxed, or in a sub-interpreter. There ought to be a way to restrict what banner ads or font loaders can do to my page.

Re: Some analysis of the 1M most popular sites on the web

#64

Post by pyre »

anantzoid wrote: post #31
jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

Minifying your JS and CSS files is a very good practice as it's not only secure, but also is compact. Grunt is a very powerful tool that does this.

As an example, on a previous project all of the javascript libraries + the app concatenated together at 3MB. Minification with uglify reduced the size to 1.5MB and using Gzip compression further reduced the transfer size to ~800K.

Re: Some analysis of the 1M most popular sites on the web

#66

Post by andrewljohnson »

jacquesm wrote: post #57
andrewljohnson wrote: post #53

Earlier quoted context omitted.

No, the solutions are far from identical to the user. In Jacques' universe, websites are more buttoned up, and less feature-ful. Every non-banking, non-critical website is taking worthless security steps in secur-e-verse, and hurting their product. In the real universe, developers of new fluff websites focus on features and user experience, grow successful, and attract many users. Developers who takes Jacques approac…

Explain to me like I'm five what features a website that hosts it's own javascript can't have versus one that loads those same javascripts from remote source?

It can't have the features that would have been built, in the time spent learning about and implementing security.

I regard nearly all security for startup-class, low-user, and low-value companies to be premature optimization, which is deadly to a new project's potential.

Re: Some analysis of the 1M most popular sites on the web

#67

Post by jacquesm »

corford wrote: post #62
jacquesm wrote: post #37

Earlier quoted context omitted.

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

What about the common trick of serving static assets via a separate domain (one that's still owned and hosted by you) to avoid the overhead of cookies from the main site being included in asset requests?

That's a good point. That would require some more work to rule out, for instance by doing a whois lookup to see if the domains have the same controlling entity.

Re: Some analysis of the 1M most popular sites on the web

#68

Post by corford »

jacquesm wrote: post #10
leeoniya wrote: post #6

Earlier quoted context omitted.

wow! i think there should be a place on the internet for publicly naming and shaming such practices. like a Darwin Award or Razzies [1] of webdev. [1] https://en.wikipedia.org/wiki/Golden_Raspberry_Awards

That's one of the things I'm considering right now. To re-write the top 1000 or so with annotations and then to sort them by category as well as an example of a site that is 'clean' in the same category. There are a ton of offenders and some of them are very well known. One of the interesting things you find when you look at this data is that the bigger sites really do have their stuff set up better (for instance, by…

If you want a never ending list of shocking offenders, be sure to check out most airline sites. They really are abysmal.

Re: Some analysis of the 1M most popular sites on the web

#69

Post by heyalexej »

jacquesm wrote: post #9
heyalexej wrote: post #7

This is very interesting. Will you release the data and code at some point?

Yes, I will definitely release the code and the dataset required to bootstrap the rest. It takes a long long time to run and you'll need a good bit of bandwidth. I won't be releasing the raw data because there is simply too much of it.

Sorry for bugging you. Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?

I crawl large-ish websites (most recently https://code.google.com with 1.8MM repos) often and am really looking forward to your dataset & code.

[1] http://phantomjs.org/api/webpage/handler/on-resource-receive...

Re: Some analysis of the 1M most popular sites on the web

#70

Post by johncolanduoni »

jacquesm wrote: post #37
gingerlime wrote: post #21

I wonder what's considered external though. If I compile / minify my javascript and CSS and then use a CDN to cache or host it -- is this considered external?? If so, how is it different from trusting my hosting provider to host my site in the first place, or my domain provider for resolving for me? How can this analysis know whether or not the resource is external? based on dns records alone? because I can still use…

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

I understand that it isn't possible to check if the external assets are hosted on a CDN bucket which is under the control of the website (but under a different domain name), but without the ability to discriminate such cases it makes your statistics on externally hosted content pretty meaningless.
Post reply on HN