Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deprecating our AJAX crawling scheme

googlewebmastercentral.blogspot.com

Post reply on HN
Print view View original

31–40 of 80 posts

Re: Deprecating our AJAX crawling scheme

#31

Post by vbezhenar »

mey wrote: post #10
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

That was my first reaction as well. "We've engineered a competitive advantage so why don't you throw out that hard work the helps our competitors." I'm not sure where I sit on this, developers who want to be noticed by other engines will continue to focus on SEO, but how many engineers care about SEO that isn't Google?

Honestly, optimizing sites for search is just wrong. It's happening now, because search is not perfect and developers have to work around its imperfectness. But in the ideal future, web masters must design websites for users, not for search engines, in the first and only place. That's what's happening now and it's good sign.

Of course Google competitors must work hard. I don't see why that's a bad thing. It's not like Bing or Yandex are going to disappear in the foreseeable future.

Re: Deprecating our AJAX crawling scheme

#32

Post by nerfhammer »

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

> your own BrowserBot

http://phantomjs.org/

Re: Deprecating our AJAX crawling scheme

#33

Post by vbezhenar »

rcconf wrote: post #29

This might be obvious to anyone who has done SEO, but can Googlebot index React/Angular websites accurately? I was always under the impression that the isomorphic aspect of React helped with SEO (not just load times.)

If a modern browser can render your site accurately, then Google can index it.

Re: Deprecating our AJAX crawling scheme

#34

Post by suneilp »

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

PhantomJS allows you to render a page and fully manipulate or search it. It's a headless WebKit browser you can use from the command line and it works pretty well. Google is obviously doing the same thing. They even used to show images of what a url looks like in the search results. They stopped doing that as I suspect it uses up a lot of resources of many sites.

Re: Deprecating our AJAX crawling scheme

#35

Post by leviathan »

mynameisvlad wrote: post #26
thephyber wrote: post #20

Earlier quoted context omitted.

The hints were littered everywhere that they did this. Google does malware detection. Not on every crawl, but a certain percentage of crawls. At my old social network site, they detected malware that must have come from ad/tracking networks because those pages had no UGC. This suggests they were using Windows virtual machines (among others) and very likely using browsers other than a heavily modified curl / wget and…

> They have long had measures in place to detect invisible text (eg. white text on white background) or hidden text (where HTML elements are styled over other HTML elements). This suggests both CSS rendering and JS rendering. No, this actually suggests it's not doing either. Both invisible and hidden text the way you've described it would be implemented with a CSS style. Not using that style would mean the text would…

The parent is talking about them penalizing sites that use such hidden text that would normally show to the crawler but be invisible to an actual human looking at the page.

Re: Deprecating our AJAX crawling scheme

#36

Post by iwilliams »

hanniabu wrote: post #6
iwilliams wrote: post #4

Earlier quoted context omitted.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Sorry if this is a stupid question as this is outside my field of work, but how can you tell if your page has been successfully indexed or not?

Just search on google for "site:http://www.yoursite.com" and you can see the pages it has indexed.

Re: Deprecating our AJAX crawling scheme

#37

Post by eurokc98 »

Gary Illyes @goog said this was happening Q1 this year, and like others mentioned lots of other direct/indirect signals have pointed this way.

http://searchengineland.com/google-may-discontinue-ajax-craw... March 5th: Gary said you may see a blog post at the Google Webmaster Blog as soon as next week announcing the decommissioning of these guidelines.

Pure speculation but interesting... The timing may have something to do with Wix, a Google Domains partner, who is having difficulty with their customer sites being indexed. The support thread shows a lot of talk around "we are following Google's Ajax guidelines so this must be a problem with Google". John Mueller is active in that thread so it's not out of the realm of possibility someone was asked to make a stronger public statement. http://searchengineland.com/google-working-on-fixing-problem...

Re: Deprecating our AJAX crawling scheme

#38

Post by iwilliams »

a2tech wrote: post #7
iwilliams wrote: post #4

Earlier quoted context omitted.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

Do you know if they sent Google a sitemap? Our client is insisting on a sitemap that has pointers to every-single-product. Something on the order of 2MM+ product pages. It seems like a bit much to me

The site is built on Tumblr which automatically generates a sitemap for the individual posts, but not any other pages on the site. For example the "about" page is not in the sitemap, but is still indexed.

Re: Deprecating our AJAX crawling scheme

#39

Post by gdulli »

cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

So they're using headless browsers. Why can't anyone do that?

Re: Deprecating our AJAX crawling scheme

#40

Post by anon1mous »

rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

The XHTML+XSLT+XML-FO stack produced pages that took 3x-10x less CPU to render. But that's dead of course.
Post reply on HN