Live data from Hacker News

Some analysis of the 1M most popular sites on the web

jacquesmattheij.com

41–50 of 130 posts

Re: Some analysis of the 1M most popular sites on the web

#41
post #33

> The request for the code contains a referring url which tells the entity hosting the script who is visiting your pages and which pages they are visiting (this goes for all externally hosted content (fonts, images etc), not just javascript) This can now be mitigated thanks to Referrer Policy [0]: "The simplest policy is No Referrer, which specifies that no referrer information is to be sent along with requests made…

Im impressed with the amount of browser support this has already. Thanks for the info.

Re: Some analysis of the 1M most popular sites on the web

#42
post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

I remember reading about http2 that using a single javascript file is an anti-pattern since http2 has smarter management of requests and can deal with them in a more granular manner.

That'll matter when anyone's actually requesting or serving pages using that protocol.

Re: Some analysis of the 1M most popular sites on the web

#43
post #39

Earlier quoted context omitted.

Yes, homepages and all the content subsequently loaded (directly or indirectly through multiple layers of scripting or iframes). Essentially what you'd get if you were to visit each and every homepage on the top list and logged the urls that were loaded as a consequence of that.

As much as I hate Flash, I don't think you can infer that then. Does your analytics consider https://www.youtube.com/ using Flash? I see no Flash on it here.

If it's not on the homepage then it would not consider it using flash. The analysis was run on the homepages, not on all the pages in those websites. (And that would require a lot more work on my part and likely would not change the results all that much).

I believe overall flash usage on the web is now about 10%, but larger sites are generally much better at keeping their sites up-to-date and to follow trends.

Advertising is another good indicator. The typical trick nowadays is to check if flash is installed using some javascript or header inspection and only to serve it up if support has been detected.

Websites that categorically include flash are the ones that were detected.

That's a good point though, I should update the text to that effect.

edit: ok, updated the text to be much more precise about flash usage and the conditions of the crawl which will lead to under-representation of flash.

Re: Some analysis of the 1M most popular sites on the web

#45
post #26

Earlier quoted context omitted.

How would the closure compiler figure out what bits and pieces of the library are triggered from the html portion of the site? (I can see how it can track the javascript bits but unless your site is entirely generated from js you'd have to start with the html)

You annotate methods in Google-JS-Closure with @public, @protected, and @private in comments. Public methods get unmangled symbols. Everything else gets renamed to a short name to save bandwidth. Dependencies are specified with goog.require. Anything that doesn't get required with goog.require or isn't called by a public function gets culled.

You can also define a separate externs file with a list of symbols that shouldn't get mangled; this is useful if eg. you're using a third-party library that you don't want to modify.

Re: Some analysis of the 1M most popular sites on the web

#46
post #38

jacquesm, maybe you didn't want to wade into the details too much, but you didn't mention a major attack vector on third party scripts, namely, the transparent caches run by nearly all ISPs. Also, unless a third party script is served over HTTPS to users, regularly verifying the scripts is useless since _your_ ISP will give you _their_ cached copy, and similar is true for all site users. Transparent CDN's are another…

The examples given were just examples, I can see a lot more possibilities beyond the ones mentioned in the article but to be honest I had not thought about the ISP caches.

Re: Some analysis of the 1M most popular sites on the web

#47
some relatively serious questions on the methodology:

- how did you define third party assets vs domain-managed assets? Is anything not hosted under example.com automatically third party? What about Twitter.com and t.co? I know this one is picky but would like a feel for the figures.

- how deep did you scrape the (million!) sites? If it's front page or similar Inwould not be surprised to see figures revised upwards significantly - once off the beaten track of even major sites the number of "let this one slide" decisions spikes a lot.

- how long did polling a million sites take?! What was the setup you used - very interested even if it has nothing to do with methodology :-)

Thank you - you have at least made me rethink my lack of blockers

Re: Some analysis of the 1M most popular sites on the web

#48
post #8

What about services like npm that distribute code? Are these analogous or do they have additional security in place?

Isn't that server side?

With dependency resolution and node_modules folders dozens of levels deep, it's pretty difficult to verify untrusted code hasn't been injected somewhere.

Re: Some analysis of the 1M most popular sites on the web

#49

This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. "If you have to use externally hosted resources such as javascript libraries then at a minimum you should verify regularly that the code has not changed " No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. Let someone e…

> This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level.

That's fine, it's only your users after all.

> No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes.

If you can fix a hole until that hole is plugged in a more permanent fashion then I think that you should.

> FTFY "A safe, but impractical and productivity-destroying aproach..."

Entirely practical and with minimal impact on your productivity. If you can find the time and expend the effort to build a product to begin with then you can certainly find the time and expend the effort to make a conscious decision about things like this.

It matters enough to me that it affects my choices with respect to which parties I deal with on the internet and I sure hope I'm not the only person that thinks like that.

And every time some user gets their day ruined by some formerly javascript widget hosting domain that got transfered to a new and malicious owner there is a bit more evidence to support that that approach is the right one.

From the end users point of view as long as it works the two solutions are identical and if you're serving up megabytes of content anyway what's the big deal about serving up some scripts directly as well?

Re: Some analysis of the 1M most popular sites on the web

#50

This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. "If you have to use externally hosted resources such as javascript libraries then at a minimum you should verify regularly that the code has not changed " No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. Let someone e…

Users assume the sites are safe and not malicious or open to attack.

There are an enormous number of implicit contracts between us as developers and our users that are not written down but are still "stuff that matters to users".

Ask a bank auditor if they think security holes does not count as "stuff that matters". Then try selling your services to said banks. It matters

Post reply on HN