iirc this is why they originally started development on what is now Chrome.
Does Google crawl dynamic content?
21–30 of 62 posts
Re: Does Google crawl dynamic content?
#22The next test could be: Does google crawl hidden text (display:none, very small, very transparent colored text)?
My guess is they do crawl it because it can have legitimate uses, but if there is to much of them on a page then they give it a lower ranking.
Re: Does Google crawl dynamic content?
#23My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.
Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.
I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.
Re: Does Google crawl dynamic content?
#24I wonder if you could use this to find information about the google crawler. Inject system and browser info into the page. Then you can find out what kind of browser engine it runs, with which settings etc.. If you wanted, you could use this information to do undetectable masking (I don't think it would work in the long run, though)
It would be also interesting to see what timeouts it still allows. I wouldn't be surprized if the modified browser "virtualizes" time and runs window.setTimeout immediately. Maybe you could make a busy loop and find out what the real timeouts are. It seems there got to be some, otherwise this would open a way to DOS the crawler (not that I'd do that).
Re: Does Google crawl dynamic content?
#25I'd really love it if you repeated the same tests for Bing, just to get coverage. (Yahoo/Baidu would be the other big two.) Historically, Bing hasn't used fully functional headless browsers to crawl, which has limited its ability to index dynamic content like this. Google has "only" 70% market share, so it seems irresponsible to make engineering decisions without testing the others. Google+Bing+Yahoo+Baidu get you to…
I just did this page. The page is indexed. When I look for the search term "Update this was posted to Google on Friday the 17th of July, 2015. Monday, the 20th" the page is shown.
Trying to find any of the other search strings in the article for the different loading variants does not return any results. So no variant of javascript injected content is working on Bing currently.
Re: Does Google crawl dynamic content?
#26Regarding SPA-based websites, as long as your site has only a few pages, these results are relevant. I would like to see the same kind of test on a site with 1000+ pages for example. I already did this kind of test in the past and it was failing miserably (i.e. only a dozen of pages were correctly indexed).
Re: Does Google crawl dynamic content?
#27My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.
Probably also one of the reasons to start the Chrome project, though their headless variant isn't open source. Are there other headless browser beside PhantomJS? PhantomJS is based on webkit (Safari).
Chromium is Open Source and embeddable, so my guess is a big part of why their headless variant isn't open source is simply that there isn't really much to it.
Re: Does Google crawl dynamic content?
#28We built our site, https://appapp.io (a search engine for the App Store) as a one page app. It serves no dynamic content in html from the server, so we were unsure to what extent google would spider/index it. As far as we can tell, it makes no difference from if it was generated server side: https://www.google.com/search?q=site%3Aappapp.io So yes, Google definitely does index dynamic content. I would love to know if…
If you do a search with 'site:site:https://appapp.io' and go to the last page of results, you'll see Google indexed officially approximately 120 results. For example, this request does not return any content: 'site:https://appapp.io "Release notes for version 6.6.0"'. It should return the page //app/we-heart-it/539124565.
Re: Does Google crawl dynamic content?
#29My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.
+1
Also, maybe it's complementary to the headed version that lot of persons use and reports back. That millions of persons requesting and rendering html, are some kind of distributed indexers too.
Re: Does Google crawl dynamic content?
#30I'm curious how google strongly penalizes SPAs for being slow to load.
The content may be indexed, but if your visitors are on a mobile network, that initial visit (or a visit with stale cache) is going to be crappy. It's great that they can read in they content (though bing cannot), but if it's buried on page two, does it even matter?
As someone who is a proponent of web perf, these kind of articles make me worried that server side rendering will be ignored because "SEO works now for Javascript", even if it's slow and google is only 70% desktop & 80% mobile search.