11–20 of 31 posts
On September 4, 1998 the Google automated network crawling system saw it's conception. By May 2011 over one billion people were dependent on it. It was growing at a geometric rate. Some time during May 2012 the Google bot cloud network began to crawl dynamic content. The growth became exponential. On August 29 of the same year, the first indications of self-awareness were spotted by a lonely hacker in Sweden. The ope…
I have a dynamic site that I'm pretty sure google does not run the javascript. How do I get google to run the javascript?
On September 4, 1998 the Google automated network crawling system saw it's conception. By May 2011 over one billion people were dependent on it. It was growing at a geometric rate. Some time during May 2012 the Google bot cloud network began to crawl dynamic content. The growth became exponential. On August 29 of the same year, the first indications of self-awareness were spotted by a lonely hacker in Sweden. The ope…
This isn't Reddit.
Maybe the easiest way to get the screencapturing browser to display a part of the page is to simulate a click. Or something.
http://ipullrank.com/googlebot-is-chrome/
There's no real proof for this of course, but it makes this change to the Google bot's behaviour make sense and explains Google's massive investment of programmer effort into Chrome and everything surrounding it (e.g. WebKit/Chromium, V8, Chrome's update mechanism).
Let's see how wide-spread this GoogleBot behavior is.
(edit) The earliest I see it pulling Ajax entry points on my sites is March 8th. It is accessing only some of the ajax'd content and the total number of these requests is ~20 times less than those for escaped_fragments.
A while back a blog post popped up here arguing that Chrome is a repackaging of a new Google bot: that Chrome was developed first as a crawler, then later repurposed as a desktop browser. http://ipullrank.com/googlebot-is-chrome/ There's no real proof for this of course, but it makes this change to the Google bot's behaviour make sense and explains Google's massive investment of programmer effort into Chrome and ever…
For those who are interested, there was a follow up to the article here: http://www.distilled.net/blog/seo/google-stop-playing-the-ji...
And Dan Clarke did some independent tests here: http://www.danclarkie.co.uk/can-the-googlebot-read-javascrip...
This was all back in Oct - Dec of '11. Basically we learned that Googlebot handles JavaScript and AJAX pretty much like a browser.
When it comes to AJAX, it appears to index the content under the destination URL of the XHR in some cases, while indexing it as part of the page making the XHR in other instances. Something about the way the AJAX request is made causes Google to treat it like a 302 redirect at times.
Standard JS window.location redirects also appear to be treated as equivalent to 302 redirects.
@dsl - I suspect you're correct. The Google Toolbar, Chrome's Opt-In Program, The Search Quality Program, and now Google Analytics Data (since the TOS change) are probably all being used to train the behavior of Googlebot when interacting with elements on a page.
Google also has plenty of patents related to computer vision, and their self-driving car is road-worthy... so processing DOM renders of the page ala Firefox's 3D View/Tilt is probably small potatoes for them.
Is this new? Several years ago I wrote an Adsense-esque ad service for use by a group of entrepreneurs that wanted to promote each other's sites. I found that Google was crawling those urls even then. The text of the ads was in an HTML file, but the actual ads were served through JavaScript.
In your case, I'd suspect they were simply following the src of your: markup... though if you read the articles cited, we suspect they've been crawling and understanding JavaScript for a pretty long time now.
Googlebot not only executing javascript on the page but also making POST requests as a result of AJAX calls.