Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deprecating our AJAX crawling scheme

googlewebmastercentral.blogspot.com

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Deprecating our AJAX crawling scheme

#51

Post by thoop »

nailer wrote: post #45

Currently I use prerender.io and this meta tag: I don't actually use #! URLs, (or pushstate, though I might use pushstate in the future) but without both of these Google can't see anything JS generated - using Google Webmaster Tools to check. Does this announcement mean I can remove the tag and stop using prerender.io now?

If Google Webmaster Tools is unable to render your website correctly, then that's a good indicator that Googlebot won't be able to render the pages correctly either. If you remove the fragment meta tag, then Google will need to render your javascript to see the page. Let us know how that goes if you try it! todd@prerender.io

Re: Deprecating our AJAX crawling scheme

#52

Post by thoop »

rgbrgb wrote: post #50
nailer wrote: post #45

Currently I use prerender.io and this meta tag: I don't actually use #! URLs, (or pushstate, though I might use pushstate in the future) but without both of these Google can't see anything JS generated - using Google Webmaster Tools to check. Does this announcement mean I can remove the tag and stop using prerender.io now?

We have a similar setup and were wondering the same thing (though we use push state). Today we were actually trying to figure out a workaround for 502s and 504s that google crawler was seeing from prerender. We just took the plunge and removed the meta tag because over 99% of our organic search traffic is from google. Fingers crossed!

I'd love to help here if I can. I'd also love to hear the results of you removing the meta tag! todd@prerender.io

Re: Deprecating our AJAX crawling scheme

#53

Post by comex »

wahnfrieden wrote: post #42
gdulli wrote: post #39

Earlier quoted context omitted.

So they're using headless browsers. Why can't anyone do that?

Scale

And perhaps security. I wouldn't be surprised if Google avoided standard C++/JIT browser engines in favor of something custom entirely written in a safe language - but if they don't, it wouldn't be that hard to get code execution on (a sandboxed portion of) Googlebot. Same goes for competitors - I don't think the state of public safe-language browsers is that good, though I'm not sure.

Re: Deprecating our AJAX crawling scheme

#54

Post by gildas »

iwilliams wrote: post #4
a2tech wrote: post #2

This is good-one of my current projects for a customer is entirely AJAX/JS rendered and we were worried that Googlebot would have a fit with it.

We recently built a site for a customer in Ember and their SEO guys were concerned about indexing. I wasn't sure how it was going to work out, but in the end Google has been able to index every page no problem.

How many pages does the site have?

Re: Deprecating our AJAX crawling scheme

#55

Post by walshemj »

thoop wrote: post #49
m0th87 wrote: post #30

Don't believe the hype. Google has been saying that they can execute javascript for years. Meanwhile, as far as I can see, most non-trivial applications still aren't being crawled successfully, including my company's. We recently got rid of prerender because of the promise from the last article from google saying the same thing [1]. It didn't work. 1: http://googlewebmastercentral.blogspot.com/2014/05/understan...

Todd from Prerender.io here. We've seen the same thing with people switching to AngularJS assuming it will work and then coming to us after they had the same issue. [1] This image is from 2014, when Google previously announced they were crawling JavaScript websites, showing our customer's switch to an AngularJS app in September. Google basically stopped crawling their website when Google was required to execute the J…

Interesting will have to reinvestigate this

Re: Deprecating our AJAX crawling scheme

#56

Post by laumars »

thephyber wrote: post #20
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

The hints were littered everywhere that they did this. Google does malware detection. Not on every crawl, but a certain percentage of crawls. At my old social network site, they detected malware that must have come from ad/tracking networks because those pages had no UGC. This suggests they were using Windows virtual machines (among others) and very likely using browsers other than a heavily modified curl / wget and…

> At my old social network site, they detected malware that must have come from ad/tracking networks because those pages had no UGC. This suggests they were using Windows virtual machines (among others) and very likely using browsers other than a heavily modified curl / wget and a headless Chrome.

I think you're making a number of wild assumptions there. You can scan and detect malware without running Windows; and there's a whole gulf of different technologies between running desktop browsers and running a modified version of curl.

With regards to your browser point, normally I'd probably suggest that Google would be running node and making use of their own V8 Javascript engine to headlessly render the pages. However Google have the resources to build something much more bespoke so I think it would be foolish of me to make blind assumptions given how little I actually know about their internal technology.

Re: Deprecating our AJAX crawling scheme

#58

Post by qopp »

comex wrote: post #53
wahnfrieden wrote: post #42

Earlier quoted context omitted.

Scale

And perhaps security. I wouldn't be surprised if Google avoided standard C++/JIT browser engines in favor of something custom entirely written in a safe language - but if they don't, it wouldn't be that hard to get code execution on (a sandboxed portion of) Googlebot. Same goes for competitors - I don't think the state of public safe-language browsers is that good, though I'm not sure.

They are probably using virtual machines anyway, so it's not hard to set it up to simply load ram state for each new page they are crawling. This sidesteps the security issue (as long as there's no sandbox escapes).

It's possible they are using components from Google Chrome as others mentioned, like V8.

Re: Deprecating our AJAX crawling scheme

#59

Post by mixonic »

m0th87 wrote: post #30

Don't believe the hype. Google has been saying that they can execute javascript for years. Meanwhile, as far as I can see, most non-trivial applications still aren't being crawled successfully, including my company's. We recently got rid of prerender because of the promise from the last article from google saying the same thing [1]. It didn't work. 1: http://googlewebmastercentral.blogspot.com/2014/05/understan...

I noticed our companies Ember.js based SPA site not being indexed well until I added a sitemap. Then it quickly appeared in the rankings.

Historically Google has been using some fork of Chrome 10 when indexing. I'm unsure what impact that is having on the reliability of app rendering, but I also trust the Google search team has done reasonable checks ensuring common sites and frameworks render correctly.

I strongly suggest using a sitemap for JS rendered sites, based on my own experience.

Re: Deprecating our AJAX crawling scheme

#60

Post by juliend2 »

rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

I think they might mitigate the need to crawl _every_ page of every web site in that fashion. They must be doing some sort of analysis to "old-school-crawl" pages that don't need javascript interpretation.
Post reply on HN