Why the assumption that "GoogleBot" is a single thing? Of course we know that google has a headless browser running, we see it's output in the instant previews, but I'm sure they still do plenty of standard crawling (and probably some half way partial JS execution and/heuristics too).
Google's Indexing Javascript more than we thought
31–35 of 35 posts
Re: Google's Indexing Javascript more than we thought
#32It occurs to me that if GoogleBot is executing client javascript you could take advantage of Google's resources for computational tasks. For instance, let me introduce you to SETI@GoogleBot. SETI@GoogleBot is much like SETI@home except it takes advantage of GoogleBot's recently discovered capabilities. Including the SETI@GoogleBot script into your web pages will cause (after the page load event) the page to fetch a c…
I imagine they run ephemeral, sandboxed chrome instances in a cgroup that limits resources like memory and cpu time. The chrome comic back when it was first launched announced it had headless capabilities, and they've gotten better at sandboxing since.
Re: Google's Indexing Javascript more than we thought
#33Earlier quoted context omitted.
One would assume that they are clever enough to have built in safeguards to prevent anything going too long, or using too much processing power.
Not just that, I would also assume that PageRank will penalise your site if a JS takes so long to execute.
Better to say "that Google will penalize your site". PageRank is a calculation on the link graph of the web, and nothing else.
Re: Google's Indexing Javascript more than we thought
#34Earlier quoted context omitted.
> I am not saying all the bots are smart, but my mantra since then has been that "if there is value for the bots to be smart, they can get very smart". I was once actually on the other side of the fence as you were, around 5-6 years ago (in a different industry, though). You're right, if there's value to be gained by scrapping other people's pages there's almost always a way round the obstacles. I remember the day wh…
Sure, automated browser testing is a whole industry, and I think we all know those tools aren't always used for testing sites you control. Take a look at Selenium and Watir.
And phantom.js
Re: Google's Indexing Javascript more than we thought
#35Earlier quoted context omitted.
The Google Webmaster Tools site performance is being measured through multiple data points which can include: - People on dial up (yes these still exist) - People in other countries, if you have not taken care of cashing or CDN, your website might load slower for far-far away visitors This represents an average across all the data points being measured. The way the data is being captured, is through: - Google Toolbar…
I guess disallowing won't work if what you say is correct i.e. Google doesn't measure site performance with Googlebot. However, this wouldn't explain the slowdown since September unless changes were also made to include JS execution time in site performance within the services you mentioned. Perhaps a solution then would be to trigger the AJAX on mouse over, but that seems kludgy. In my case, I need make the AJAXed c…
It really depends on what you are trying to accomplish if you need to worry what Google is reporting with respect to the page load time.
In general, it's always good to pay attention to page load times, regardless what Google finds of it!
You can experiment with Asynchronous calls, or slow load jQuery scripts, which kick off after the headers and html framework already have been loaded.
Overall, I would not worry about the reports in Google WMC that much, just try to get faster in general.
If you are serious in delivering an ultra speedy service online, there are services which can test your site or application on multiple locations, different OS and connection speed or using a different browser. But these services are pricy, trust me on that one!