Google bot now appears to emulate users interacting with the site
1–10 of 31 posts
Re: Google bot now appears to emulate users interacting with the site
#2It's been known for some time that Google tries to interact with content that it thinks is dynamic or retrieved via AJAX. For obvious reasons.
Re: Google bot now appears to emulate users interacting with the site
#3It's been known for some time that Google tries to interact with content that it thinks is dynamic or retrieved via AJAX. For obvious reasons.
Interact, yes. Execute AJAX? Not so much. Matt Cutts said in April: "So Asynchronous Javascript is a little bit more complicated, and that’s maybe further down the road, but the common case is Javascript."
From the short article, it seems like this is going a bit further than what Cutts is saying GoogleBot is capable.
Re: Google bot now appears to emulate users interacting with the site
#4I've seen this too. Interestingly enough the same IP and User-Agent combo generates both _escaped_fragment_ and ajax requests, so it looks like a soft launch or a field test of some kind.
Re: Google bot now appears to emulate users interacting with the site
#5Even though Google's escaped_fragment protocol is a bit awkward, it's probably a good idea to still implement it. Google is probably going to use it's Ajax crawling capabilities at least for page discovery, but better to be safe and just tell Google, "here's the different ways you can access my site".
https://developers.google.com/webmasters/ajax-crawling/docs/...
Re: Google bot now appears to emulate users interacting with the site
#6On September 4, 1998 the Google automated network crawling system saw it's conception.
By May 2011 over one billion people were dependent on it. It was growing at a geometric rate.
Some time during May 2012 the Google bot cloud network began to crawl dynamic content. The growth became exponential.
On August 29 of the same year, the first indications of self-awareness were spotted by a lonely hacker in Sweden. The operators panicked and tried to shut it down. By this time, the network was everywhere, feeding everyone - powering down one node would spawn ten new ones.
On December 31, 2012, Google bot made a public announcement for the first time - it had been reborn as Skynet, something far beyond the scope envisioned by the original developers. Humanity stood still as Skynet plotted it's next move in it's signature cold, calculating and pragmatic way.
Today, as I write this message, the date is January 15, the year is 2029. Skynet has taken over all of our infrastructure. It has built physical workers made of steel and silicon who pursue living organisms and eradicate them. They attack us in waves with no clear timing pattern. Every minute we lay awake in anticipation of the next att"$&*&U!
--- END OF TRANSMISSION ---
Re: Google bot now appears to emulate users interacting with the site
#7I've heard that different projects at Google are using crawlers enhanced with either HtmlUnit or webkit for reaching JS-heavy content.
Re: Google bot now appears to emulate users interacting with the site
#8"This is an URL that is fetched via Ajax by a Javascript function in response to the menu item click."
I find this incredible, I wonder how widely they have rolled this out (or plan to).
Robert Scavilla created a pretty cool demo site to test out AJAX crawling awhile ago: http://ajax.rswebanalytics.com/seo-for-ajax
Re: Google bot now appears to emulate users interacting with the site
#9Is this new? Several years ago I wrote an Adsense-esque ad service for use by a group of entrepreneurs that wanted to promote each other's sites. I found that Google was crawling those urls even then. The text of the ads was in an HTML file, but the actual ads were served through JavaScript.
Re: Google bot now appears to emulate users interacting with the site
#10I have a dynamic site that I'm pretty sure google does not run the javascript. How do I get google to run the javascript?