Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deprecating our AJAX crawling scheme

googlewebmastercentral.blogspot.com

Post reply on HN
Print view View original

71–80 of 80 posts

Re: Deprecating our AJAX crawling scheme

#71

Post by tracker1 »

gukov wrote: post #28
rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

I'm wondering if Google is somehow, in some way, using the rendering data generated by the Chrome clients and/or Android to aid with processing power it takes to index everything.

More likely they're getting lots of data from analytics users for a great number of sites as it is, and only really need to do custom renders for load time analysis for some sites, and not necessarily all pages... to a larger extent, I'm pretty sure they could have an optimized rendering pipeline for a headless chrome that actually works better than, by comparison, phantomjs.

Re: Deprecating our AJAX crawling scheme

#72

Post by tracker1 »

juliend2 wrote: post #60
rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

I think they might mitigate the need to crawl _every_ page of every web site in that fashion. They must be doing some sort of analysis to "old-school-crawl" pages that don't need javascript interpretation.

What if they don't actually "render" the dom as part of the "load" analysis... this means they don't necessarily need to handle certain UI/UX aspects that can be bypassed.. they could then output the "rendered" content for passthrough to the same system that does their general crawl analysis for additional details.

The work could be broken up in any number of ways... from my own testing, and experience with others testing. Content crawls/recrawls from JS data tends to lag a couple days behind initial scan... having an updating sitemap xml resource is a good idea for "new" content if you're doing JS based content.. also, rescans will still lag well behind the general non-js content scans...

Re: Deprecating our AJAX crawling scheme

#73

Post by tracker1 »

greglindahl wrote: post #44
chc wrote: post #21

Earlier quoted context omitted.

By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites. Do you have some reason to doubt that?

If crawling with JS costs 1,000X or 10,000X as much as crawling without, it's fair to say that even Google isn't going to crawl 100s of billions of pages executing JS. As a former web-scale search engine CTO, my opinions are commonly surprising to folks who haven't built a web-scale crawler/search engine.

My own experiments/experience shows that recrawls happen about 1/3 as often and tend to lag a few days behind for JS content vs inlined/delivered content. It's helped a little by dynamically delivering the sitemap data, but even that only speeds things up a little.

My guess is they're putting about 1/10th the effort into keeping things freshly indexed for JS, but may well be devoting 2x the resources vs directly received content.

Re: Deprecating our AJAX crawling scheme

#74

Post by tracker1 »

devNoise wrote: post #17
a2tech wrote: post #2

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

About a year ago I wrote a post[1] about how I couldn't get google to index my AngularJS app. My main problem was the interaction between googlebot and the S3 server. I'll have to go back and test if the crawler's behavior will render the correct content. 1 - https://medium.com/@devNoise/seo-fail-figuring-out-why-i-can...

Do you have a sitemap.xml for common routes.. also is your angular app actually doing routing (hash based or push state)?

Re: Deprecating our AJAX crawling scheme

#75

Post by tracker1 »

nostrademons wrote: post #43
eurokc98 wrote: post #37

Gary Illyes @goog said this was happening Q1 this year, and like others mentioned lots of other direct/indirect signals have pointed this way. http://searchengineland.com/google-may-discontinue-ajax-craw... March 5th: Gary said you may see a blog post at the Google Webmaster Blog as soon as next week announcing the decommissioning of these guidelines. Pure speculation but interesting... The timing may have something…

I'm betting that they finally solved the scalability problems with headless WebKit. Google's been able to index JS since about 2010, but when I left in 2014, you couldn't rely on this for anything but the extreme head of the site distribution because they could only run WebKit/V8 on a limited subset of sites with the resources they had available. Either they got a whole bunch more machines devoted to indexing or they…

I'd say both are pretty likely.. another round of lower-power servers with potentially more cores... more infrastructure... Combined with improvements in headless rendering pipelines. I haven't looked into it in well over a year now, but last I checked dynamic updates took about 2-3 days to get discovered vs. server-delivered being hours for a relatively popular site.

I'm guessing they've likely cut this time in half through a combination of additional resources, and performance improvements. Wondering if they'd be willing to push this out as something better than PhantomJS... probably not as it's a pretty big competative advantage.

I know MS has been doing JS rendering for a few years, they show up in analytics traffic (big time if you change your routing scheme on a site with lots of routes, will throw off your numbers).

Re: Deprecating our AJAX crawling scheme

#76

Post by tracker1 »

vbezhenar wrote: post #33
rcconf wrote: post #29

This might be obvious to anyone who has done SEO, but can Googlebot index React/Angular websites accurately? I was always under the impression that the isomorphic aspect of React helped with SEO (not just load times.)

If a modern browser can render your site accurately, then Google can index it.

It's always lagged in my experience... I'm hoping this announcement means that lag is under a day instead of the 2-3 it was a bit over a year ago.

Re: Deprecating our AJAX crawling scheme

#77

Post by tracker1 »

espeed wrote: post #24

This was the missing piece for Polymer elements / custom web components. Now that Google has confirmed it's indexing JavaScript, web-component adoption should take off.

I want to like polymer/web-components... I just find that it kind of flips around the application controls that redux+react offers. I'm not sure that I like it better in practice.

Re: Deprecating our AJAX crawling scheme

#78

Post by grey-area »

thoop wrote: post #49
m0th87 wrote: post #30

Don't believe the hype. Google has been saying that they can execute javascript for years. Meanwhile, as far as I can see, most non-trivial applications still aren't being crawled successfully, including my company's. We recently got rid of prerender because of the promise from the last article from google saying the same thing [1]. It didn't work. 1: http://googlewebmastercentral.blogspot.com/2014/05/understan...

Todd from Prerender.io here. We've seen the same thing with people switching to AngularJS assuming it will work and then coming to us after they had the same issue. [1] This image is from 2014, when Google previously announced they were crawling JavaScript websites, showing our customer's switch to an AngularJS app in September. Google basically stopped crawling their website when Google was required to execute the J…

Perhaps this could be down to response times too, they might crawl much quicker if given static HTML very quickly?

What were the page render times for the two types of page?

Re: Deprecating our AJAX crawling scheme

#79

Post by spacecowboy_lon »

mey wrote: post #10
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

That was my first reaction as well. "We've engineered a competitive advantage so why don't you throw out that hard work the helps our competitors." I'm not sure where I sit on this, developers who want to be noticed by other engines will continue to focus on SEO, but how many engineers care about SEO that isn't Google?

Depends what locale's you are targeting I am doing some strategy proposals for a client to help with their move into Asia.

Biaudu is one SE that doesn't crawl JS well from my research.

Re: Deprecating our AJAX crawling scheme

#80

Post by devNoise »

tracker1 wrote: post #74
devNoise wrote: post #17

Earlier quoted context omitted.

About a year ago I wrote a post[1] about how I couldn't get google to index my AngularJS app. My main problem was the interaction between googlebot and the S3 server. I'll have to go back and test if the crawler's behavior will render the correct content. 1 - https://medium.com/@devNoise/seo-fail-figuring-out-why-i-can...

Do you have a sitemap.xml for common routes.. also is your angular app actually doing routing (hash based or push state)?

I have the .html5Mode set to true so it was routing based on push state.

I hadn't gotten around to creating a process to generate the sitemap.xml before I gave up on the site. For SEO, we were more concerned with getting the time sensitive content indexed.

Post reply on HN