Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does Google crawl dynamic content?

centrical.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Does Google crawl dynamic content?

#41

Post by jacquesm »

kawsper wrote: post #38
jacquesm wrote: post #36

Earlier quoted context omitted.

I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.

Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.

No, I haven't. Just chrome for the one and gmail for the other.

The way it works is simple, I made a random url, stuck a script in there that sends an email with the url as the subject header. The first script I visited using chrome, the second script I mailed myself a link to from another email account to a gmail account. Both scripts fired when chrome activated the links so I know they work, then I simply let it rest.

I guess the gmail one would require re-crawling of all gmail for it to fire, the chrome only one would be dependent on the version of chrome that I ran the test on phoning home with the link, and I did not re-try this for every version of chrome (or on every platform). So it's not a perfect method but it definitely puts the lie to chrome or gmail data being directly used to power google search results that people would expect to remain confidential because only they have the urls. Score one for obscurity and nice of google to ignore these paths.

I could set up another url for a test using the DNS but you're free to do so yourself as well of course, it is definitely an interesting idea.

And if either of those scripts ever does fire I'll rip google a new one, that would be the sort of abuse of trust that gets my temperature up. But for now I'll take the fact that it hasn't happened as proof that google can be trusted with data to some extent.

Re: Does Google crawl dynamic content?

#42

Post by cm2187 »

HappyTypist wrote: post #11
Rangi42 wrote: post #4

Earlier quoted context omitted.

This is a problem with a lot of paginated sites, such as Tumblr, various forums, and comment pages. Anything that's ordered from newest to oldest won't have a constant correspondence between URL and content.

That's why pagination (when newest to oldest) should be designed as something like "after_id=x". Sure, there is some implementation complexity, but your users will love you if they can actually find the content they searched for.

I am of the opinion that for text, pagination should be avoided. Except if the only aim is to increase the page refresh for advertising purposes.

Re: Does Google crawl dynamic content?

#43

Post by dansingerman »

gildas wrote: post #37
dansingerman wrote: post #35

Earlier quoted context omitted.

Yes, we have more work to do to get Google to index all our content. It's still a bit of a mystery to us (Google webmaster tools tells us about 2500 pages are in their index) Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...

Moreover, note that in the SERP, you can find some pages with the good title but with the wrong description, cf. the 2 first results for example here http://imgur.com/cOV85UJ . BTW, if you don't want every app to be indexed, I would recommend you the tag in the app pages. Alternatively, you could define the canonical URL as the URL of the original content.

Thanks. We have a lot of optimisation to do. When we launched we weren't sure to what extent we'd be indexed.

Now that we know we are (by Google at least) we can put some focus on optimising our SERPS.

Re: Does Google crawl dynamic content?

#44

Post by neogenix »

gildas wrote: post #37
dansingerman wrote: post #35

Earlier quoted context omitted.

Yes, we have more work to do to get Google to index all our content. It's still a bit of a mystery to us (Google webmaster tools tells us about 2500 pages are in their index) Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...

Moreover, note that in the SERP, you can find some pages with the good title but with the wrong description, cf. the 2 first results for example here http://imgur.com/cOV85UJ . BTW, if you don't want every app to be indexed, I would recommend you the tag in the app pages. Alternatively, you could define the canonical URL as the URL of the original content.

We have a ReactJs application served from S3 with dynamic content on https://teletext.io. All content (eg texts) is loaded asynchronously from a CloudFront distribution and I can confirm this is indexed properly by Google (see https://www.google.com/search?q=site%3Ateletext.io).

The only thing we cannot seem to get right are the meta title and meta description. If you set that asynchronously based on the React page you are rendering, Google only seems to pick it up in about 10% of the pages. So the SERPS doesn't look as pretty as you would like. I didn't found a solution for that yet. :-(

Re: Does Google crawl dynamic content?

#45

Post by jlgaddis »

BHSPitMonkey wrote: post #32
frik wrote: post #20

Earlier quoted context omitted.

Imagine a blog. On frontpage example.com/ (= example.com/?page=1) it shows the newest 10 articles, on example.com/?page=2 it shows the next 10 articles, and so on. Every article has the actual URL in its headline hyperlink (e.g. example.com/?article=123) Now imagine that Google links to example.com/?page=2 as it found the search phrase also there (at a given time only Google knows). So when the user clicks on the sea…

I think you missed the part earlier in this thread about using ?after=id rather than ?page=n. See Reddit for an example.

You can see it here on HN as well. Click "new" at the top of this page, then "More", and see the "next=" parameter in the URL.

Re: Does Google crawl dynamic content?

#46

Post by _ao789 »

roboshake wrote: post #10

Google may be indexing dynamic content now, but the question I'm curious about is how it affects crawl efficiency. I can't imagine indexing JS content is as efficient as indexing content returned from the original HTTP request.

They just wait around for the page to change a bit and once it's done (after some timeout) then it takes what's currently on the DOM as being the actual page. This is then sent back, analysed and parsed as the content. Similar to how phantomjs does page rendering with timeout..

Re: Does Google crawl dynamic content?

#47

Post by Aissen »

kawsper wrote: post #23
porker wrote: post #18

Earlier quoted context omitted.

Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.

I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.

They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.

Re: Does Google crawl dynamic content?

#48

Post by anarchitect »

porker wrote: post #18
jimrandomh wrote: post #7

My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.

Are they also using people's browsing history to 'find' content? E.g. from their safety filter? Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.

We (kingandmcgaw.com) have seen plenty of examples that suggest Googlebot tries to discover pages that might exist (as distinct from crawling interlinking pages).

This includes navigating the URL structure (in finding blah.com/one/two/three.html there may be attempts to /one/two and /one) and password-protected admins which are not linked to from anywhere (we suspect Analytics or toolbars are telling Google these pages exist). As a result, Googlebot generates a bunch of false positives in our error logs.

Re: Does Google crawl dynamic content?

#50

Post by sandercentrical »

mywacaday wrote: post #39

centrical.com is blocked where i work by mcafee web gateway due to GTI reputation identifing it as malicious and high risk

Ha! Interesting. It is my personal website since 2001 or so, and has never been compromised. It is a flat file website in S3 with Cloudflare on top - not much to hack. I think mcAfee is a tiny bit too strict ;-)
Post reply on HN