Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's Indexing Javascript more than we thought

distilled.net

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Google's Indexing Javascript more than we thought

#21

Post by eli »

paganel wrote: post #19
bowyakka wrote: post #14

I have been saying this for years, but most people have refused to believe me Like most people a long time ago I also held the belief that robots were just dumb scripts, however I learnt that this is not the case when I had to trap said robots for a previous employer. See at the time I was working for one of the many online travel sites; now most people probably are not aware that there is quite a bit of money that c…

> I am not saying all the bots are smart, but my mantra since then has been that "if there is value for the bots to be smart, they can get very smart". I was once actually on the other side of the fence as you were, around 5-6 years ago (in a different industry, though). You're right, if there's value to be gained by scrapping other people's pages there's almost always a way round the obstacles. I remember the day wh…

Sure, automated browser testing is a whole industry, and I think we all know those tools aren't always used for testing sites you control.

Take a look at Selenium and Watir.

Re: Google's Indexing Javascript more than we thought

#22

Post by wahnfrieden »

chrisguitarguy wrote: post #3
wiradikusuma wrote: post #2

i wonder if it also means we don't need to implement _escaped_fragment_ anymore http://code.google.com/web/ajaxcrawling/docs/getting-started...

Just because GoogleBot can crawl and execute/index javascript, doesn't mean that it will on your site. The best bet would be to keep them. Or take them off and see what happens. If you don't see negative effects, then you will have discovered something interesting.

We sure saw a hell of a lot of improvement when we moved a lot of client-side rendering to the server. Before that, Google wasn't indexing any of our content that was rendered in JS.

We know googlebot executes JS. But this could be primarily for things like validating that a site isn't cheating by dynamically hiding search keywords and so on. It is also for generating the page preview.

It's good to see that they're starting to index JS-rendered content too, as seen with the Facebook comments widget, but it does not mean we're free to ignore these issues just yet. As it stands, client-side rendering in general means a huge hit to search ranking and experience (e.g. your search results won't have anything meaningful to say, and will probably use irrelevant text like your static footer/copyright notice as the description).

Re: Google's Indexing Javascript more than we thought

#23

Post by wahnfrieden »

eli wrote: post #20

I don't buy this argument. Wanting to have a more complete rendering engine for their crawler might have been a factor in designing Chrome, but I can't imagine it was in any way the driving force. The costs of developing a browser that runs well on millions of different computers and configurations are far beyond what it would take to make a really great headless version of WebKit for your crawler.

Yeah, I took that part of the original article (the one this one is citing) to be a dumbed-down explanation of googlebot's new behaviors. That article's audience were SEO people, not "engineers". It's unfortunately misleading enough that we get articles like these once you try to extrapolate from that.

Re: Google's Indexing Javascript more than we thought

#25

Post by thenextcorner »

cr4zy wrote: post #16
xtacy wrote: post #13

Earlier quoted context omitted.

Not just that, I would also assume that PageRank will penalise your site if a JS takes so long to execute.

Looking at Google Webmaster tools, I see a significant decline in my reported site performance starting in September, even though my site's speed has improved significantly since then by my own measures. Assuming this is due to our 'next' feature that AJAXs in, I'm going to disallow the 'next' urls in robots.txt and cross my fingers.

The Google Webmaster Tools site performance is being measured through multiple data points which can include: - People on dial up (yes these still exist) - People in other countries, if you have not taken care of cashing or CDN, your website might load slower for far-far away visitors This represents an average across all the data points being measured.

The way the data is being captured, is through: - Google Toolbar - Google Chrome - Google DNS services

Your observations of the speed and the improvements are not always similar as the ones from the aggregated data Google has access to.

I'm not sure what you are trying to accomplish with the dis-allowment of the next urls in robots.txt. Can you explain more what your hypothesis is in this test, and how you would measure success?

Re: Google's Indexing Javascript more than we thought

#28

Post by jQueryIsAwesome »

nyellin wrote: post #24

You might be able to check what the Googlebot executes by adding javascript to your site and checking the thumbnail. EDIT: Removed comment about the bot's user-agent. The article links to a Google FAQ which answers the question.

They execute absolutely everything you put in Javascript; it looks exactly as it looks in Chrome. And it looks like it takes the snapshot after all the initial processing is done.

Javascript-heavy site with perfect snapshots: http://goo.gl/xNUIM

But they also index and take a snapshot of the no-javascript version: http://goo.gl/eP84M

Re: Google's Indexing Javascript more than we thought

#29

Post by [deleted] »

heyitsnick wrote: post #7

Is there any example of a site having their dynamically generated* disqus comments indexed by google? Disqus is probably one of the most common form of ajax-generated content on the web, so if this were the case that googlebot was actively indexing dynamic content like this, I would expect to see disqus supported. * disqus has an API to allow you to display disqus comments serverside, so some disqus implementations -…

[deleted]

Re: Google's Indexing Javascript more than we thought

#30

Post by cr4zy »

thenextcorner wrote: post #25
cr4zy wrote: post #16

Earlier quoted context omitted.

Looking at Google Webmaster tools, I see a significant decline in my reported site performance starting in September, even though my site's speed has improved significantly since then by my own measures. Assuming this is due to our 'next' feature that AJAXs in, I'm going to disallow the 'next' urls in robots.txt and cross my fingers.

The Google Webmaster Tools site performance is being measured through multiple data points which can include: - People on dial up (yes these still exist) - People in other countries, if you have not taken care of cashing or CDN, your website might load slower for far-far away visitors This represents an average across all the data points being measured. The way the data is being captured, is through: - Google Toolbar…

I guess disallowing won't work if what you say is correct i.e. Google doesn't measure site performance with Googlebot. However, this wouldn't explain the slowdown since September unless changes were also made to include JS execution time in site performance within the services you mentioned.

Perhaps a solution then would be to trigger the AJAX on mouse over, but that seems kludgy. In my case, I need make the AJAXed content part of the initial page load anyway, for the sake of user experience. But I can see cases where Google should not be counting AJAX as part of the page load time. God forbid somebody uses long polling for example. Maybe Google is doing this in a smart way, looking at the changes after the AJAX and determining if they should count as part of the page load.

Post reply on HN