Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Some analysis of the 1M most popular sites on the web

jacquesmattheij.com

Post reply on HN
Print view View original

51–60 of 130 posts

Re: Some analysis of the 1M most popular sites on the web

#51

Post by nostrademons »

Semiapies wrote: post #42
vvanders wrote: post #14

Earlier quoted context omitted.

I remember reading about http2 that using a single javascript file is an anti-pattern since http2 has smarter management of requests and can deal with them in a more granular manner.

That'll matter when anyone's actually requesting or serving pages using that protocol.

Chrome, Firefox, and Opera all support it, as do Google, Twitter, Akamai, Jetty, Apache, and several others:

https://github.com/http2/http2-spec/wiki/Implementations

https://en.wikipedia.org/wiki/HTTP/2

That's a big chunk of the Internet right there. IE 11 and Safari 9 both support it, so once their respective betas go public that's the rest of the client-side support. Nginx is supposed to support it by the end of the year; once that happens most sites will get it just by tweaking a config file:

https://www.nginx.com/blog/how-nginx-plans-to-support-http2/

Re: Some analysis of the 1M most popular sites on the web

#52

Post by jacquesm »

lifeisstillgood wrote: post #47

some relatively serious questions on the methodology: - how did you define third party assets vs domain-managed assets? Is anything not hosted under example.com automatically third party? What about Twitter.com and t.co? I know this one is picky but would like a feel for the figures. - how deep did you scrape the (million!) sites? If it's front page or similar Inwould not be surprised to see figures revised upwards s…

I will release code + data for bootstrapping but until then here are my answers to your questions:

> how did you define third party assets vs domain-managed assets? Is anything not hosted under example.com automatically third party? What about Twitter.com and t.co? I know this one is picky but would like a feel for the figures.

That's based on the hosting domain being the same or a superset of the domain that the page originally came from.

> how deep did you scrape the (million!) sites?

Just the homepage.

> If it's front page or similar Inwould not be surprised to see figures revised upwards significantly - once off the beaten track of even major sites the number of "let this one slide" decisions spikes a lot.

That's true.

> how long did polling a million sites take?!

20 days. About 50K sites per day which significantly cramped my ability to do other work here.

> What was the setup you used - very interested even if it has nothing to do with methodology :-)

A simple laptop with 16G of ram and a regular (spinning) drive on a 200/20 cable connection. 40 worker threads concurrently with a simple php script to supervise the crawler and another script to do the analysis.

Most of the data was discarded right after crawling a page, only the URLS that were loaded as a result of loading the homepage were kept as well as the mime type of the result.

Re: Some analysis of the 1M most popular sites on the web

#53

Post by andrewljohnson »

jacquesm wrote: post #49
andrewljohnson wrote: post #18

This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. "If you have to use externally hosted resources such as javascript libraries then at a minimum you should verify regularly that the code has not changed " No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. Let someone e…

> This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. That's fine, it's only your users after all. > No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. If you can fix a hole until that hole is plugged in a more permanent fashion then I think that you should. > FTFY "A sa…

No, the solutions are far from identical to the user.

In Jacques' universe, websites are more buttoned up, and less feature-ful. Every non-banking, non-critical website is taking worthless security steps in secur-e-verse, and hurting their product.

In the real universe, developers of new fluff websites focus on features and user experience, grow successful, and attract many users. Developers who takes Jacques approach build websites that fail and never get used by anyone.

And also no, no, no.... no one is like you. No one cares if developers do what you propose, no one avoids websites that don't.

Re: Some analysis of the 1M most popular sites on the web

#54

Post by Glyptodon »

andrewljohnson wrote: post #18

This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. "If you have to use externally hosted resources such as javascript libraries then at a minimum you should verify regularly that the code has not changed " No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. Let someone e…

I hope you don't develop any websites I use.

Re: Some analysis of the 1M most popular sites on the web

#55

Post by hexasquid »

jacquesm wrote: post #11
ktusznio wrote: post #8

What about services like npm that distribute code? Are these analogous or do they have additional security in place?

Isn't that server side?

Not really. NPM is also used with a tool called browserify to enable frontend web developers to use NPM modules in the browser.

Re: Some analysis of the 1M most popular sites on the web

#57

Post by jacquesm »

andrewljohnson wrote: post #53
jacquesm wrote: post #49

Earlier quoted context omitted.

> This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. That's fine, it's only your users after all. > No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. If you can fix a hole until that hole is plugged in a more permanent fashion then I think that you should. > FTFY "A sa…

No, the solutions are far from identical to the user. In Jacques' universe, websites are more buttoned up, and less feature-ful. Every non-banking, non-critical website is taking worthless security steps in secur-e-verse, and hurting their product. In the real universe, developers of new fluff websites focus on features and user experience, grow successful, and attract many users. Developers who takes Jacques approac…

Explain to me like I'm five what features a website that hosts it's own javascript can't have versus one that loads those same javascripts from remote source?

Re: Some analysis of the 1M most popular sites on the web

#59

Post by lifeisstillgood »

larrys wrote: post #25
andrewljohnson wrote: post #18

This article should be read as ranty research, not practical advice. It'd be fine to fix these issues, but not at the website developer level. "If you have to use externally hosted resources such as javascript libraries then at a minimum you should verify regularly that the code has not changed " No, you shouldn't. You should focus on stuff that matters to users, not existential internet security holes. Let someone e…

Agree the advice is well intentioned and is correct (in theory according to what I read) but not entirely practical. For example: "then at a minimum you should verify regularly that the code has not changed (you have to hope that you are looking at the same code that your users see)" Who exactly is the "you" in the above statement and who pays the "you" money to fix this and keep on top of it on an ongoing basis? And…

>> Who exactly is the "you" in the above statement and who pays the "you" money to fix this and keep on top of it on an ongoing basis? And for how long?

This is not a negative! This is an upwelling opportunity for a retainer. I know someone whose business is warranting other agencies sites - he patches holes in a site they built but don't see the percentage in maintaining - he is doing well enough to hire in new folks.

I was doing hashes of jquery certainly five years ago and I suspect earlier - it's one of those rings that seems obvious in the buildscript.

Re: Some analysis of the 1M most popular sites on the web

#60

Post by pyre »

jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

> These days using them is pretty much pointless and incurs a performance penalty, yet everybody still uses them.

Would you rather than when (e.g.) there is a security patch for OpenSSL, that you have to wait for all software using OpenSSL to deploy updates? Or would you rather that one update to OpenSSL (likely from your OS vendor) fixes all of the software depending on it?

Edit: People seem to be commenting to this through the lense of CDNs and JavaScript, but the sentence previous to the one I quoted was:

> This is somewhat similar to the situation we have with operating systems: we created shared libraries to save disk space and memory.

Which is not talking about CDNs and JavaScript, but shared libraries on your desktop. I'm not saying that all usage of shared libraries is valid. I'm just saying that to toss out the concept as entirely useless (and having no redeeming value) in a modern setting varies from the truth.

Post reply on HN