31–40 of 62 posts
The next test could be: Does google crawl hidden text (display:none, very small, very transparent colored text)? My guess is they do crawl it because it can have legitimate uses, but if there is to much of them on a page then they give it a lower ranking.
Earlier quoted context omitted.
What about a redirect from your home page to whatever page 1 is at the moment?
Imagine a blog. On frontpage example.com/ (= example.com/?page=1) it shows the newest 10 articles, on example.com/?page=2 it shows the next 10 articles, and so on. Every article has the actual URL in its headline hyperlink (e.g. example.com/?article=123) Now imagine that Google links to example.com/?page=2 as it found the search phrase also there (at a given time only Google knows). So when the user clicks on the sea…
http://searchengineland.com/tested-googlebot-crawls-javascri...
Earlier quoted context omitted.
Probably also one of the reasons to start the Chrome project, though their headless variant isn't open source. Are there other headless browser beside PhantomJS? PhantomJS is based on webkit (Safari).
Chromium is Open Source and embeddable, so my guess is a big part of why their headless variant isn't open source is simply that there isn't really much to it.
So a crawler based on a headless browser that consumes little memory and runs for weeks is a major achivement.
We built our site, https://appapp.io (a search engine for the App Store) as a one page app. It serves no dynamic content in html from the server, so we were unsure to what extent google would spider/index it. As far as we can tell, it makes no difference from if it was generated server side: https://www.google.com/search?q=site%3Aappapp.io So yes, Google definitely does index dynamic content. I would love to know if…
If you do a search with 'site:site: https://appapp.io' and go to the last page of results, you'll see Google indexed officially approximately 120 results. For example, this request does not return any content: 'site: https://appapp.io "Release notes for version 6.6.0"'. It should return the page / /app/we-heart-it/539124565.
Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...
My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.
Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.
Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.
Earlier quoted context omitted.
If you do a search with 'site:site: https://appapp.io' and go to the last page of results, you'll see Google indexed officially approximately 120 results. For example, this request does not return any content: 'site: https://appapp.io "Release notes for version 6.6.0"'. It should return the page / /app/we-heart-it/539124565.
Yes, we have more work to do to get Google to index all our content. It's still a bit of a mystery to us (Google webmaster tools tells us about 2500 pages are in their index) Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...
BTW, if you don't want every app to be indexed, I would recommend you the tag in the app pages. Alternatively, you could define the canonical URL as the URL of the original content.
Earlier quoted context omitted.
Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.
I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.
> So, very soon, the days of pre-rendering PhantomJs snapshots and serving shadow content to spiders will be over.
To be clear: webmasters of sites with dynamic content should not celebrate yet. There are still influential spiders other than Google's that do not parse JavaScript (for example, Facebook[1] and Twitter[2]).
[1] https://developers.facebook.com/docs/sharing/webmasters/craw...
[2] Can't find an official statement on this, but https://twittercommunity.com/search?q=javascript%20crawl