Does Google crawl dynamic content?

#1

Post by pstadler » Mon, Dec 07, 2015, 10:51 AM UTC

Does Google crawl dynamic content?
centrical.com

Re: Does Google crawl dynamic content?

#2

Post by largote » Tue, Dec 08, 2015, 3:49 AM UTC

TL;DR: Yes, most of the time.

Re: Does Google crawl dynamic content?

#3

Post by userbinator » Tue, Dec 08, 2015, 4:00 AM UTC

I suppose this explains all the times I've seen a promising search result with the words I was searching for prominently highlighted, then visited the page to find what I was looking for is no longer there. Sometimes the cached, text-only version has it, and sometimes not. Alternatively, I'll see search results with none of the words I was searching for, yet perhaps they did sometime in the past. Rather annoying.

Re: Does Google crawl dynamic content?

#4

Post by Rangi42 » Tue, Dec 08, 2015, 4:25 AM UTC

userbinator wrote: post #3

I suppose this explains all the times I've seen a promising search result with the words I was searching for prominently highlighted, then visited the page to find what I was looking for is no longer there. Sometimes the cached, text-only version has it, and sometimes not. Alternatively, I'll see search results with none of the words I was searching for, yet perhaps they did sometime in the past. Rather annoying.

This is a problem with a lot of paginated sites, such as Tumblr, various forums, and comment pages. Anything that's ordered from newest to oldest won't have a constant correspondence between URL and content.

Re: Does Google crawl dynamic content?

#5

Post by dyoo1979 » Tue, Dec 08, 2015, 5:23 AM UTC

Probably relevant: http://googlewebmastercentral.blogspot.com/2014/05/understan...

Re: Does Google crawl dynamic content?

#6

Post by andreasklinger » Tue, Dec 08, 2015, 5:35 AM UTC

I am sure that google "discovers" javascript/ajax content. They also mention this on their guides several times.

But are there any experiments/results related to SEO impact/crawl frequency etc?

Re: Does Google crawl dynamic content?

#7

Post by jimrandomh » Tue, Dec 08, 2015, 5:35 AM UTC

My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.

Re: Does Google crawl dynamic content?

#8

Post by m0dest » Tue, Dec 08, 2015, 6:26 AM UTC

I'd really love it if you repeated the same tests for Bing, just to get coverage. (Yahoo/Baidu would be the other big two.) Historically, Bing hasn't used fully functional headless browsers to crawl, which has limited its ability to index dynamic content like this.

Google has "only" 70% market share, so it seems irresponsible to make engineering decisions without testing the others. Google+Bing+Yahoo+Baidu get you to 98%.

Re: Does Google crawl dynamic content?

#9

Post by frik » Tue, Dec 08, 2015, 6:31 AM UTC

jimrandomh wrote: post #7

My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.

Probably also one of the reasons to start the Chrome project, though their headless variant isn't open source.

Are there other headless browser beside PhantomJS? PhantomJS is based on webkit (Safari).

Re: Does Google crawl dynamic content?

#10

Post by roboshake » Tue, Dec 08, 2015, 6:45 AM UTC

Google may be indexing dynamic content now, but the question I'm curious about is how it affects crawl efficiency. I can't imagine indexing JS content is as efficient as indexing content returned from the original HTTP request.