Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does Google execute JavaScript?

stephanboyer.com

Post reply on HN
Print view View original

21–30 of 114 posts

Re: Does Google execute JavaScript?

#21

Post by codedokode »

Google executes JS but maybe not on every website. If you have a JS error reporting tool on such site then you can get reports from Google IP addresses. I saw them first maybe 4 or 5 years ago.

Executing JS everywhere would require a lot of CPU time and I think Google prefers not to do that when possible. And indexing a JS app is a very complicated task anyway (it is difficult for a robot to even find navigation elements if they are implemented as div's with onclick handler instead of links) so you better use sitemaps to make sure the bot can find content.

And I don't think it is necessary to index rich apps. It makes no sense to index a ticket search app (the data become outdated too fast) or an online spreadsheet editor. Just make indexable pages as server-rendered HTML pages and put their URLs into a sitemap.

Also Google looks for strings in JS code that look like URLS (e.g var url = '/some/page') and crawls them later.

Re: Does Google execute JavaScript?

#22

Post by icebraining »

amorphid wrote: post #14

One random thought... Google goes to SomeWebsite.com. The site has only enough HTML to load a big ol' JavaScript app, which Google slowly crawls. Well, that JS app makes a bunch of AJAX calls. There's no reason I can think of that would prevent Google from remembering which AJAX calls were made, and then just crawling the URLs for those calls on subsequent visits. Why load SomeWebsite.com's JavaScript.com every time…

You still have to run the JS on the original page to know which text loaded from the JSON actually gets shown. You might have N pages loading the same file but showing different things.

Re: Does Google execute JavaScript?

#23

Post by codedokode »

amorphid wrote: post #14

One random thought... Google goes to SomeWebsite.com. The site has only enough HTML to load a big ol' JavaScript app, which Google slowly crawls. Well, that JS app makes a bunch of AJAX calls. There's no reason I can think of that would prevent Google from remembering which AJAX calls were made, and then just crawling the URLs for those calls on subsequent visits. Why load SomeWebsite.com's JavaScript.com every time…

Crawling AJAX data alone makes no sense because it could be just a piece of JSON and Google needs a rendered HTML page with an URL it can show in the results. If you have some data that are not available at a separate URL (e.g. they are loaded when user presses a button), they will not be indexed.

Re: Does Google execute JavaScript?

#24

Post by codedokode »

jwatte wrote: post #3

My pet theory is that part of the anonymous usage data Chrome sends back, is digested page contents that go into pagerank. And such browser level digesting would be on rendered pages (after JavaScript execution.) I have no other reason to believe it is true other than it's what I would do to distribute the job of crawling the web to my users if i were Google :-)

I think that is unlikely because then SEO specialists would generate and send such data to improve the positions of their websites.

But Google could use this to discover new URLs that are not linked anywhere.

Re: Does Google execute JavaScript?

#25

Post by ebbv »

nrjdhsbsid wrote: post #9

My pet theory is that Google actually developed chrome as a web crawler and that consumer release was to ensure that Google would always be able to crawl pages (since sites would always want to work properly with chrome) It also explains why they effectively killed flash and Java applets. They were competing technologies that weren't owned by Google and not crawlable. If they would have taken off, Google's position a…

Google did not kill Flash. Chrome came bundled with Flash already installed. Apple made the biggest push to kill Flash and once support for HTML5 video was solid and widespread the main use case for Flash was gone.

Java applets just always sucked and only masochists/sadists ever used them.

Re: Does Google execute JavaScript?

#26

Post by a3_nm »

codedokode wrote: post #24
jwatte wrote: post #3

My pet theory is that part of the anonymous usage data Chrome sends back, is digested page contents that go into pagerank. And such browser level digesting would be on rendered pages (after JavaScript execution.) I have no other reason to believe it is true other than it's what I would do to distribute the job of crawling the web to my users if i were Google :-)

I think that is unlikely because then SEO specialists would generate and send such data to improve the positions of their websites. But Google could use this to discover new URLs that are not linked anywhere.

Some URLs are private by design, so I think it would be an awfully bad idea to discover new URLs in this way.

Re: Does Google execute JavaScript?

#27

Post by samsonradu »

neurostimulant wrote: post #12
Fogest wrote: post #8

Earlier quoted context omitted.

Yeah, it would make sense. There has to be more reason to giving people a free web browser other than just that it uses Google search.

Chrome does send domains/urls entered in the omnibar to google (I think someone did the experiment to test it several years ago), but sending out page content to google? If it were true it'll cause a huge legal and privacy problem, especially in places with tight privacy law like Europe.

Exactly, imagine if Google sends the page content after you log into your bank account.

Re: Does Google execute JavaScript?

#28

Post by mootothemax »

I'm convinced that Google has several Googlebots that are run depending on how popular a site is.

That is, new and low traffic sites are crawled by less intelligent bots, and as the site gets more visitors or better rankings, more complicated and resource intensive bots are deployed.

How this might work with the most popular sites out there, the Amazons and Wikipedias of this world - I'm not so sure about that. If I were in charge, I'd be tempted to have customised bots and ranking weights for each of these exceptional sites.

Sadly the chances of getting a real answer on this in my lifetime are close to zero.

Re: Does Google execute JavaScript?

#30

Post by GoToRO »

My experience is that content hidden behind js will get indexed later, if at all, and it will be updated not as often. Also they will run js for bigger sites first, and not so much for smaller sites.
Post reply on HN