Some analysis of the 1M most popular sites on the web • Hacker News Board

Re: Some analysis of the 1M most popular sites on the web

#71

Post by jacquesm » Fri, Jul 24, 2015, 6:24 PM UTC

andrewljohnson wrote: post #66

jacquesm wrote: post #57

Earlier quoted context omitted.

Explain to me like I'm five what features a website that hosts it's own javascript can't have versus one that loads those same javascripts from remote source?

It can't have the features that would have been built, in the time spent learning about and implementing security. I regard nearly all security for startup-class, low-user, and low-value companies to be premature optimization, which is deadly to a new project's potential.

> I regard nearly all security for startup-class, user-less, and low-value companies to be premature optimization.

I can't see anybody working on user-less websites anyway but I sincerely hope that you'll make it plain which start-ups you work for so I can avoid them. Security and abuse potential are very important for start-ups because you have only one reputation and if you lose that you're pretty much done for.

I can point you to several pretty harsh reminders of how start-ups that don't take end-user security serious can end up.

Re: Some analysis of the 1M most popular sites on the web

#72

Post by bcg1 » Fri, Jul 24, 2015, 6:26 PM UTC

pradn wrote: post #36

jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

Can you please elaborate how using share libraries is "pretty much pointless and incurs a performance penalty"? That goes against my intuition of how they work.

Think it means that most machines are not memory or disk constrained these days, but there is extra processing to perform the dynamic linking. Sort of a cost-benefit argument it seems. Doesn't address the issue of security etc and the benefit of just having 1 instance of a library to update when maintaining a complete system however.

Re: Some analysis of the 1M most popular sites on the web

#73

Post by Semiapies » Fri, Jul 24, 2015, 6:28 PM UTC

nostrademons wrote: post #51

Semiapies wrote: post #42

Earlier quoted context omitted.

That'll matter when anyone's actually requesting or serving pages using that protocol.

Chrome, Firefox, and Opera all support it, as do Google, Twitter, Akamai, Jetty, Apache, and several others: https://github.com/http2/http2-spec/wiki/Implementations https://en.wikipedia.org/wiki/HTTP/2 That's a big chunk of the Internet right there. IE 11 and Safari 9 both support it, so once their respective betas go public that's the rest of the client-side support. Nginx is supposed to support it by the end of th…

Only for TLS or only in beta versions. It's still going to be awhile before it's worth it to sabotage older browser performance, even once sites update their servers.

And in the end, that'll just embolden sites to crust their pages with more analytics and trackers until the performance isn't any better.

Re: Some analysis of the 1M most popular sites on the web

#74

Post by jacquesm » Fri, Jul 24, 2015, 6:29 PM UTC

throwaway41597 wrote: post #65

How deep did you crawl? I would have guessed the flash usage to be higher. How big is the dataset? How long did it take? Which tools did you use besides phantomjs? Nice job!

> How deep did you crawl?

Front pages only.

> I would have guessed the flash usage to be higher.

When adding all the pages in a site it no doubt will be. I'll update the article to clarify this.

> How big is the dataset?

In flight: huge, but after culling and keeping only the bits that I needed it was a lot smaller, about 20G.

> How long did it take?

About 10 days.

> Which tools did you use besides phantomjs?

Just some php glue scripts, nothing fancy, about 500 lines.

Re: Some analysis of the 1M most popular sites on the web

#75

Post by pjungwir » Fri, Jul 24, 2015, 6:31 PM UTC

I agree re not using externally-hosted Javascript. In fact I seem to remember a year ago Google Code having connectivity issues and jQuery all over the place failing to load. I was glad on that day that I always host my own jQuery.

Re tracking, I ran into this embedded in some webfonts CSS a project was using (downloaded from one of those font websites):

    /* @import must be at top of file, otherwise CSS will not work */
    @import url("//elided.example.com/count/35d82f");

    @font-face {font-family: 'Foo'; font-weight: 300; src: url('/webfonts/foo.eot');.....}

That @import returns nothing. It is just part of their tracking/licensing. And it was really slow! And I love the lying comment they included.

Re: Some analysis of the 1M most popular sites on the web

#76

Post by jacquesm » Fri, Jul 24, 2015, 6:32 PM UTC

johncolanduoni wrote: post #70

jacquesm wrote: post #37

Earlier quoted context omitted.

If the url used to fetch the file is not related to the domain the original html comes from then that would be counted as external. You can point *.my-domain.com to an external resource but it would see that resource as still under your control. I will post the code soon.

I understand that it isn't possible to check if the external assets are hosted on a CDN bucket which is under the control of the website (but under a different domain name), but without the ability to discriminate such cases it makes your statistics on externally hosted content pretty meaningless.

I don't agree with that. From the point of view of the user that content might as well have come from a third party since - just like the headless browser used in the testing - they have absolutely no way to verify that short of doing a bunch of whois lookups. And if there is one thing that a user should be able to verify then it is that the entity sending them the main page is the same entity as the one that sends them the rest of the stuff on that page and to refuse all or part of the transaction if that isn't the case.

After all: there is only one slot in the URL bar, which strongly suggests to the user that that is the entity they are transacting with.

What technical tricks are pulled behind the scenes have no bearing on that.

Re: Some analysis of the 1M most popular sites on the web

#77

Post by xorcist » Fri, Jul 24, 2015, 6:34 PM UTC

pyre wrote: post #60

jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

> These days using them is pretty much pointless and incurs a performance penalty, yet everybody still uses them. Would you rather than when (e.g.) there is a security patch for OpenSSL, that you have to wait for all software using OpenSSL to deploy updates? Or would you rather that one update to OpenSSL (likely from your OS vendor) fixes all of the software depending on it? Edit: People seem to be commenting to this…

Google doesn't back-port fixes to JQuery.

You can link without specifying the version number, but then you don't get full caching, so it's not common in practice.

Re: Some analysis of the 1M most popular sites on the web

#78

Post by throwaway41597 » Fri, Jul 24, 2015, 6:35 PM UTC

captainmuon wrote: post #63

One thing in this context is that it is basically impossible for a website to check the integrity of an external (js) resource without loading it. This is a consequence of the web security model. Its basically impossible to get the contents of a .js file without executing it, say for checksum verification (at least without CORS, and even with, you might trigger an additional download, I haven't tested it). But it's t…

Look SRI up on this very page.

Web pages can make requests to other origins (GET image, script, XHR, POST to iframe, XHR). CORS allows you to read the response. But what you're asking would probably be hard to transition the whole web to without too much spam and DOS'ing.

The sandboxing for an external script you want already is feasible with an iframe with a different origin.

Re: Some analysis of the 1M most popular sites on the web

#79

Post by jacquesm » Fri, Jul 24, 2015, 6:36 PM UTC

heyalexej wrote: post #69

jacquesm wrote: post #9

Earlier quoted context omitted.

Yes, I will definitely release the code and the dataset required to bootstrap the rest. It takes a long long time to run and you'll need a good bit of bandwidth. I won't be releasing the raw data because there is simply too much of it.

Sorry for bugging you. Did you store results from the response ¹ metadata object for every domain and process it later or use Regex to parse the HTML content? I crawl large-ish websites (most recently https://code.google.com with 1.8MM repos) often and am really looking forward to your dataset & code. [1] http://phantomjs.org/api/webpage/handler/on-resource-receive...

> Did you store results from the response¹ metadata object for every domain and process it later or use Regex to parse the HTML content?

That would have constrained throughput too much so I opted for culling it during the crawl to just content-type and url, this was then processed to extract the various bits of information. I did use the 'resource received' trick you linked above. Very useful.

Re: Some analysis of the 1M most popular sites on the web

#80

Post by losername » Fri, Jul 24, 2015, 6:37 PM UTC

pyre wrote: post #60

jwr wrote: post #13

Likely culprits are "performance analyzers" that grade a website and report an "F" (failing) grade for not using CDN-hosted common libraries. This is a red herring: this idea that the user will already have a cached copy of CDN-hosted jQuery is bogus. Even for a common library like jQuery: the number of versions of jQuery that are in use is likely above 50, and the number of popular CDNs that host jQuery is surely ab…

> These days using them is pretty much pointless and incurs a performance penalty, yet everybody still uses them. Would you rather than when (e.g.) there is a security patch for OpenSSL, that you have to wait for all software using OpenSSL to deploy updates? Or would you rather that one update to OpenSSL (likely from your OS vendor) fixes all of the software depending on it? Edit: People seem to be commenting to this…

> This is a red herring: this idea that the user will already have a cached copy of [open-ssl] is bogus

He says this is because of many different versions in use

While this isn't true for a managed repository of software, it is still true for most software releases so the mismatch just might happen further down the line.