Deprecating our AJAX crawling scheme • Hacker News Board

Re: Deprecating our AJAX crawling scheme

#41

Post by anon1mous » Wed, Oct 14, 2015, 9:31 PM UTC

cmg wrote: post #23

MichaelApproved wrote: post #16

Earlier quoted context omitted.

I'm wonder if they're cutting out a lot of the rendering that PhantomJS is doing. Not to say that any type of rendering is cheap but I'm guessing they have a limited version of a JS rendering engine that does just enough to index the page. I bet they'd also skip on all the FB like buttons and other common social media elements that don't impact the content.

They're starting to consider page load speed as a factor in rankings, which would lead me to believe that they're letting all the social buttons / trackers / media load.

How do you know a page has loaded? A complex page with ads, AJAX, WebSockets may be constantly busy. Most social buttons, ads, etc. are now loaded by callbacks, that usually finish after the page has rendered.

Re: Deprecating our AJAX crawling scheme

#42

Post by wahnfrieden » Wed, Oct 14, 2015, 9:35 PM UTC

gdulli wrote: post #39

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

So they're using headless browsers. Why can't anyone do that?

Scale

Re: Deprecating our AJAX crawling scheme

#43

Post by nostrademons » Wed, Oct 14, 2015, 9:35 PM UTC

eurokc98 wrote: post #37

Gary Illyes @goog said this was happening Q1 this year, and like others mentioned lots of other direct/indirect signals have pointed this way. http://searchengineland.com/google-may-discontinue-ajax-craw... March 5th: Gary said you may see a blog post at the Google Webmaster Blog as soon as next week announcing the decommissioning of these guidelines. Pure speculation but interesting... The timing may have something…

I'm betting that they finally solved the scalability problems with headless WebKit. Google's been able to index JS since about 2010, but when I left in 2014, you couldn't rely on this for anything but the extreme head of the site distribution because they could only run WebKit/V8 on a limited subset of sites with the resources they had available. Either they got a whole bunch more machines devoted to indexing or they figured out how to speed it up significantly.

Re: Deprecating our AJAX crawling scheme

#44

Post by greglindahl » Wed, Oct 14, 2015, 9:39 PM UTC

chc wrote: post #21

greglindahl wrote: post #12

Earlier quoted context omitted.

You should still be worried. Just because googlebot expensively evaluates JS for some websites doesn't mean it will evaluate JS for your brand-new website. You might get crawled a lot less deeply than if you had good content in your static pages.

By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites. Do you have some reason to doubt that?

If crawling with JS costs 1,000X or 10,000X as much as crawling without, it's fair to say that even Google isn't going to crawl 100s of billions of pages executing JS.

As a former web-scale search engine CTO, my opinions are commonly surprising to folks who haven't built a web-scale crawler/search engine.

Re: Deprecating our AJAX crawling scheme

#45

Post by nailer » Wed, Oct 14, 2015, 9:52 PM UTC

Currently I use prerender.io and this meta tag:

I don't actually use #! URLs, (or pushstate, though I might use pushstate in the future) but without both of these Google can't see anything JS generated - using Google Webmaster Tools to check.

Does this announcement mean I can remove the tag and stop using prerender.io now?

Re: Deprecating our AJAX crawling scheme

#46

Post by dragonwriter » Wed, Oct 14, 2015, 9:54 PM UTC

anon1mous wrote: post #40

rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

The XHTML+XSLT+XML-FO stack produced pages that took 3x-10x less CPU to render. But that's dead of course.

> The XHTML+XSLT+XML-FO stack produced pages that took 3x-10x less CPU to render. But that's dead of course.

Was it ever alive? I never found a decent browser XSL-FO renderer, there were some that seemed kind of proof-of-concept-ish (the only decent XSL-FO rendering I ever encountered was intended for print-like media, mostly PDF, rather than for browsing.)

Re: Deprecating our AJAX crawling scheme

#47

Post by elorant » Wed, Oct 14, 2015, 9:56 PM UTC

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

You can use one of the many headless browsers available. Selenium, phantomjs, phantomjs+casper, webkit, chromium, awesomium, name your poison. All are quite competent in rendering modern web pages. You don’t need to reinvent the wheel.

Re: Deprecating our AJAX crawling scheme

#48

Post by elorant » Wed, Oct 14, 2015, 10:03 PM UTC

rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

Chrome is much lighter than phantomjs. I use Awesomium which is a .net port of Chromium and it loads pages at half the time phantom does with much less CPU load. My guess is that Google can refine it even further.

Re: Deprecating our AJAX crawling scheme

#49

Post by thoop » Wed, Oct 14, 2015, 10:03 PM UTC

m0th87 wrote: post #30

Don't believe the hype. Google has been saying that they can execute javascript for years. Meanwhile, as far as I can see, most non-trivial applications still aren't being crawled successfully, including my company's. We recently got rid of prerender because of the promise from the last article from google saying the same thing [1]. It didn't work. 1: http://googlewebmastercentral.blogspot.com/2014/05/understan...

Todd from Prerender.io here. We've seen the same thing with people switching to AngularJS assuming it will work and then coming to us after they had the same issue.

[1] This image is from 2014, when Google previously announced they were crawling JavaScript websites, showing our customer's switch to an AngularJS app in September. Google basically stopped crawling their website when Google was required to execute the JavaScript. Once that customer implemented Prerender.io in October, everything went back to normal.

Another customer recently (June 2015) did a test for their housing website. They tested the use of Prerender.io on a portion of their site against Google rendering the JS of another portion of their site. Here are the results they sent to me:

Suburb A was prerendered and Google asked for 4,827 page impressions over 9 days Suburb B was not prerendered and Google asked for 188 page impressions over 9 days

We've actually talked to Google some about this issue to see if they could improve their crawl speed for JavaScript websites since we believe it's a good thing for Google to be able to crawl JavaScript websites correctly, but it looks like any website with a large number of pages still needs to be sceptical about getting all of their pages into Google's index correctly.

1: https://s3.amazonaws.com/prerender-static/gwt_crawl_stats.pn...

Re: Deprecating our AJAX crawling scheme

#50

Post by rgbrgb » Wed, Oct 14, 2015, 10:05 PM UTC

nailer wrote: post #45

Currently I use prerender.io and this meta tag: I don't actually use #! URLs, (or pushstate, though I might use pushstate in the future) but without both of these Google can't see anything JS generated - using Google Webmaster Tools to check. Does this announcement mean I can remove the tag and stop using prerender.io now?

We have a similar setup and were wondering the same thing (though we use push state). Today we were actually trying to figure out a workaround for 502s and 504s that google crawler was seeing from prerender. We just took the plunge and removed the meta tag because over 99% of our organic search traffic is from google. Fingers crossed!