Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does Google crawl dynamic content?

centrical.com

Post reply on HN
Print view View original

11–20 of 62 posts

Re: Does Google crawl dynamic content?

#11

Post by HappyTypist »

Rangi42 wrote: post #4
userbinator wrote: post #3

I suppose this explains all the times I've seen a promising search result with the words I was searching for prominently highlighted, then visited the page to find what I was looking for is no longer there. Sometimes the cached, text-only version has it, and sometimes not. Alternatively, I'll see search results with none of the words I was searching for, yet perhaps they did sometime in the past. Rather annoying.

This is a problem with a lot of paginated sites, such as Tumblr, various forums, and comment pages. Anything that's ordered from newest to oldest won't have a constant correspondence between URL and content.

That's why pagination (when newest to oldest) should be designed as something like "after_id=x". Sure, there is some implementation complexity, but your users will love you if they can actually find the content they searched for.

Re: Does Google crawl dynamic content?

#12

Post by gPphX »

I have modified wikipedia pages, then googled it, to see search result "instantly" updated.

Also, sneaky web sites often give different results to the googlebot user agent than to a non-google firefox user agent

https://en.wikipedia.org/wiki/User_agent

https://addons.mozilla.org/en-GB/firefox/search/?q=user+agen...

Re: Does Google crawl dynamic content?

#13

Post by dansingerman »

We built our site, https://appapp.io (a search engine for the App Store) as a one page app. It serves no dynamic content in html from the server, so we were unsure to what extent google would spider/index it.

As far as we can tell, it makes no difference from if it was generated server side: https://www.google.com/search?q=site%3Aappapp.io

So yes, Google definitely does index dynamic content. I would love to know if it ranks it equivalently.

Also, Bing does not: http://www.bing.com/search?q=site%3aappapp.io

(apologies for the minor self-promotion)

Re: Does Google crawl dynamic content?

#15

Post by anewhnaccount »

gPphX wrote: post #12

I have modified wikipedia pages, then googled it, to see search result "instantly" updated. Also, sneaky web sites often give different results to the googlebot user agent than to a non-google firefox user agent https://en.wikipedia.org/wiki/User_agent https://addons.mozilla.org/en-GB/firefox/search/?q=user+agen...

Google deliberately crawls in a non-Google looking way to try and detect "masking".

Re: Does Google crawl dynamic content?

#16

Post by frik »

HappyTypist wrote: post #11
Rangi42 wrote: post #4

Earlier quoted context omitted.

This is a problem with a lot of paginated sites, such as Tumblr, various forums, and comment pages. Anything that's ordered from newest to oldest won't have a constant correspondence between URL and content.

That's why pagination (when newest to oldest) should be designed as something like "after_id=x". Sure, there is some implementation complexity, but your users will love you if they can actually find the content they searched for.

What about the frontpage which is usually page 1 of the pagination?

Google should give the actual article URL a higher score and pagination pages a lower score. So that in their search results I see the content first and the "dupe content" on pagination pages not at all (or way down). (at least for common blog software)

Re: Does Google crawl dynamic content?

#17

Post by frik »

Offtopic: "Google search results on tablet"

Recently Google changed their search result page for tablets. First it looked fine, and useful.

But many times the first result page is now completely full of advertisements, only the second page now shows usual links to websites like Github, Wikipedia, Youtube, etc. of a common search term. Very annoying! And the Youtube link is broken on iPad (it tries to link to a non HTTP address). I am just unlucky to be part of an AB-testing?

An news article about the changes: http://searchengineland.com/google-launches-new-search-resul...

Re: Does Google crawl dynamic content?

#18

Post by porker »

jimrandomh wrote: post #7

My theory is that the Google crawler is a modified, headless version of Chrome. These results seem consistent with that hypothesis.

Are they also using people's browsing history to 'find' content? E.g. from their safety filter?

Though I don't think it's happening, I've thought it'd be very clever if users became the search spider for Google, telling them when content had gone stale and/or doing the spidering on Google's part. Just by using Google's browser.

Re: Does Google crawl dynamic content?

#19

Post by Sacho »

frik wrote: post #16
HappyTypist wrote: post #11

Earlier quoted context omitted.

That's why pagination (when newest to oldest) should be designed as something like "after_id=x". Sure, there is some implementation complexity, but your users will love you if they can actually find the content they searched for.

What about the frontpage which is usually page 1 of the pagination? Google should give the actual article URL a higher score and pagination pages a lower score. So that in their search results I see the content first and the "dupe content" on pagination pages not at all (or way down). (at least for common blog software)

What about a redirect from your home page to whatever page 1 is at the moment?

Re: Does Google crawl dynamic content?

#20

Post by frik »

Sacho wrote: post #19
frik wrote: post #16

Earlier quoted context omitted.

What about the frontpage which is usually page 1 of the pagination? Google should give the actual article URL a higher score and pagination pages a lower score. So that in their search results I see the content first and the "dupe content" on pagination pages not at all (or way down). (at least for common blog software)

What about a redirect from your home page to whatever page 1 is at the moment?

Imagine a blog. On frontpage example.com/ (= example.com/?page=1) it shows the newest 10 articles, on example.com/?page=2 it shows the next 10 articles, and so on. Every article has the actual URL in its headline hyperlink (e.g. example.com/?article=123)

Now imagine that Google links to example.com/?page=2 as it found the search phrase also there (at a given time only Google knows). So when the user clicks on the search result link that leads to example.com/?page=2 NOW should the blog software know what Google or the user wants?

One thing that comes to my mind is to use the referrer and if it's a common search engine parse the s=SEARCHTERM string and use an internal article search to find the best matching article.

Post reply on HN