Deprecating our AJAX crawling scheme

#1

Post by antichaos » Wed, Oct 14, 2015, 6:57 PM UTC

Deprecating our AJAX crawling scheme
googlewebmastercentral.blogspot.com

Re: Deprecating our AJAX crawling scheme

#2

Post by a2tech » Wed, Oct 14, 2015, 7:24 PM UTC

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

Re: Deprecating our AJAX crawling scheme

#3

Post by rdoherty » Wed, Oct 14, 2015, 7:30 PM UTC

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale.

I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

Re: Deprecating our AJAX crawling scheme

#4

Post by iwilliams » Wed, Oct 14, 2015, 7:36 PM UTC

a2tech wrote: post #2

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Re: Deprecating our AJAX crawling scheme

#5

Post by Jake232 » Wed, Oct 14, 2015, 7:39 PM UTC

rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

Can confirm. Launched a project recently with over 500 concurrent PhantomJS workers. Let's just say my hosting bill is significantly more expensive than it was.

Re: Deprecating our AJAX crawling scheme

#6

Post by hanniabu » Wed, Oct 14, 2015, 7:48 PM UTC

iwilliams wrote: post #4

a2tech wrote: post #2

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Sorry if this is a stupid question as this is outside my field of work, but how can you tell if your page has been successfully indexed or not?

Re: Deprecating our AJAX crawling scheme

#7

Post by a2tech » Wed, Oct 14, 2015, 7:49 PM UTC

iwilliams wrote: post #4

a2tech wrote: post #2

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Do you know if they sent Google a sitemap? Our client is insisting on a sitemap that has pointers to every-single-product. Something on the order of 2MM+ product pages. It seems like a bit much to me

Re: Deprecating our AJAX crawling scheme

#8

Post by cotillion » Wed, Oct 14, 2015, 7:50 PM UTC

So they're actually evaluating all js and css Googlebot is consuming. That's insane.

Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

Re: Deprecating our AJAX crawling scheme

#9

Post by acdha » Wed, Oct 14, 2015, 7:51 PM UTC

hanniabu wrote: post #6

iwilliams wrote: post #4

Earlier quoted context omitted.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Sorry if this is a stupid question as this is outside my field of work, but how can you tell if your page has been successfully indexed or not?

See https://www.google.com/webmasters/tools/ for index stats, error reports, etc.

Re: Deprecating our AJAX crawling scheme

#10

Post by mey » Wed, Oct 14, 2015, 7:54 PM UTC

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

That was my first reaction as well. "We've engineered a competitive advantage so why don't you throw out that hard work the helps our competitors."

I'm not sure where I sit on this, developers who want to be noticed by other engines will continue to focus on SEO, but how many engineers care about SEO that isn't Google?