Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does Google crawl dynamic content?

centrical.com

Post reply on HN
Print view View original

51–60 of 62 posts

Re: Does Google crawl dynamic content?

#51

Post by sandercentrical »

bigethan wrote: post #30

I'm curious how google strongly penalizes SPAs for being slow to load. The content may be indexed, but if your visitors are on a mobile network, that initial visit (or a visit with stale cache) is going to be crappy. It's great that they can read in they content (though bing cannot), but if it's buried on page two, does it even matter? As someone who is a proponent of web perf, these kind of articles make me worried…

SPAs should not be slow. If they are, they haven't been designed properly, I think.

Re: Does Google crawl dynamic content?

#52

Post by oneeyedpigeon »

peterhartree wrote: post #40

The post author writes: > So, very soon, the days of pre-rendering PhantomJs snapshots and serving shadow content to spiders will be over. To be clear: webmasters of sites with dynamic content should not celebrate yet. There are still influential spiders other than Google's that do not parse JavaScript (for example, Facebook[1] and Twitter[2]). [1] https://developers.facebook.com/docs/sharing/webmasters/craw... [2] C…

And, of course, there are still all the other reasons why you shouldn't be serving static text via javascript; I wish the article had included such a caveat.

Re: Does Google crawl dynamic content?

#53

Post by endymi0n »

dansingerman wrote: post #35
gildas wrote: post #28

Earlier quoted context omitted.

If you do a search with 'site:site: https://appapp.io' and go to the last page of results, you'll see Google indexed officially approximately 120 results. For example, this request does not return any content: 'site: https://appapp.io "Release notes for version 6.6.0"'. It should return the page / /app/we-heart-it/539124565.

Yes, we have more work to do to get Google to index all our content. It's still a bit of a mystery to us (Google webmaster tools tells us about 2500 pages are in their index) Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...

Maybe clearing up some parts about the mysteries there:

I recently did a talk at GDG about the efforts to get our SPA ranking in the Google SERPs, maybe some enlightening parts in there :)

TL;DR: All possible, same rankings, some caveats though.

GDG DevFest 2015 - We can't use Angular. It will hurt our SEO. [video]

https://www.youtube.com/watch?v=Sp1pfC1M7Dg

Re: Does Google crawl dynamic content?

#54

Post by markeroon »

endymi0n wrote: post #53
dansingerman wrote: post #35

Earlier quoted context omitted.

Yes, we have more work to do to get Google to index all our content. It's still a bit of a mystery to us (Google webmaster tools tells us about 2500 pages are in their index) Our goal is not to have every app indexed (as that will by definition be non-original content), but to have our app category pages indexed, e.g. https://appapp.io/gb/genre=Games;has_iap=false;price=Paid/se...

Maybe clearing up some parts about the mysteries there: I recently did a talk at GDG about the efforts to get our SPA ranking in the Google SERPs, maybe some enlightening parts in there :) TL;DR: All possible, same rankings, some caveats though. GDG DevFest 2015 - We can't use Angular. It will hurt our SEO. [video] https://www.youtube.com/watch?v=Sp1pfC1M7Dg

We found the same thing with Angular. Ironic and annoying.

Re: Does Google crawl dynamic content?

#55

Post by [deleted] »

markeroon wrote: post #54
endymi0n wrote: post #53

Earlier quoted context omitted.

Maybe clearing up some parts about the mysteries there: I recently did a talk at GDG about the efforts to get our SPA ranking in the Google SERPs, maybe some enlightening parts in there :) TL;DR: All possible, same rankings, some caveats though. GDG DevFest 2015 - We can't use Angular. It will hurt our SEO. [video] https://www.youtube.com/watch?v=Sp1pfC1M7Dg

We found the same thing with Angular. Ironic and annoying.

[deleted]

Re: Does Google crawl dynamic content?

#56

Post by e12e »

Aissen wrote: post #47
kawsper wrote: post #23

Earlier quoted context omitted.

I think they use their DNS service to learn about new domains, it could be interesting if it could be used for new content but I don't think so.

They don't really need to, Verisign (for .com) and ICANN (for gTLDs) provide access to the zone files.

Depends on why they'd want to know about "new" domains. For indexing content, it would certainly make sense to index stuff people (appear to be) read(ing). Just because someone registers a domain, doesn't mean it has any (interesting) content.

Re: Does Google crawl dynamic content?

#57

Post by uhoreg »

kawsper wrote: post #38
jacquesm wrote: post #36

Earlier quoted context omitted.

I can confirm that is not the case. I've had a canary page for that purpose set up for years and it never fired. If it ever does you can expect a blog post. I have another one that is set up to fire if google ever uses gmailed links to crawl, that one too never fired. Now, that's only one bit of data but if you want to be sure you can set up a trigger page of your own.

Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.

DNS only sees the host name, so it can't be used to see what URLs are being accessed.

Re: Does Google crawl dynamic content?

#58

Post by tracker1 »

anewhnaccount wrote: post #15
gPphX wrote: post #12

I have modified wikipedia pages, then googled it, to see search result "instantly" updated. Also, sneaky web sites often give different results to the googlebot user agent than to a non-google firefox user agent https://en.wikipedia.org/wiki/User_agent https://addons.mozilla.org/en-GB/firefox/search/?q=user+agen...

Google deliberately crawls in a non-Google looking way to try and detect "masking".

Yeah, that expert sex change site used to bug the hell out of me for that reason... scroll forever and a day after loading.

Re: Does Google crawl dynamic content?

#59

Post by jacquesm »

uhoreg wrote: post #57
kawsper wrote: post #38

Earlier quoted context omitted.

Have you tried accessing the site through Google DNS? 8.8.8.8 and 8.8.4.4.

DNS only sees the host name, so it can't be used to see what URLs are being accessed.

I think that was the test case that he intended. So set up a domain that is otherwise unknown and then use google's DNS and see if the domain is hit by the search engine.

Re: Does Google crawl dynamic content?

#60

Post by elliottcarlson »

m0dest wrote: post #8

I'd really love it if you repeated the same tests for Bing, just to get coverage. (Yahoo/Baidu would be the other big two.) Historically, Bing hasn't used fully functional headless browsers to crawl, which has limited its ability to index dynamic content like this. Google has "only" 70% market share, so it seems irresponsible to make engineering decisions without testing the others. Google+Bing+Yahoo+Baidu get you to…

"Originally, none of the actual web crawling and data housing was done by Yahoo! itself. In 2001, the searchable index was powered by Inktomi and later was powered by Google until 2004, when Yahoo! Search became independent. On July 29, 2009, Microsoft and Yahoo! announced a deal in which Bing would henceforth power Yahoo! Search."

https://en.wikipedia.org/wiki/Yahoo!_Search

Post reply on HN