Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

Post reply on HN
Print view View original

21–30 of 112 posts

Re: Javascript apps can be fully crawlable

#21

Post by timr »

Don't do this.

Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head.

You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since you already are, implicitly), or you're gambling that they won't bust you even though your site looks like a classic keyword stuffer.

Re: Javascript apps can be fully crawlable

#22

Post by imslavko »

RoboTeddy wrote: post #5

This looks similar to Meteor's "spiderable" package http://docs.meteor.com/#spiderable

Looks like that's exactly what Meteor's spiderable package does since 08/2012[0]: look at user-agent, run phantomjs for 10s and return a rendered page once google/facebook crawler detected.

[0]: http://www.meteor.com/blog/2012/08/08/search-engine-optimiza...

Re: Javascript apps can be fully crawlable

#23

Post by dwwoelfel »

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header:
then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page.

Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and we've been using this method at https://circleci.com for the past year.

Waiting for google to request the page with _escaped_fragment_ should also prevent you from getting penalized for slow load times or showing googlebot different content.

Re: Javascript apps can be fully crawlable

#24

Post by reissbaker »

_lex wrote: post #20
10098 wrote: post #17

Earlier quoted context omitted.

Pretty sure the load time problem can be mitigated by caching.

Best case scenario you still have network trips going out to the service, so it's still not a great solution UNLESS the caching was done by your webapp - which is what I spoke about at the end of my comment above. Unless this works w/o adding network roundtrips on each request, it's not a great idea.

I think the Unix philosophy of "do one thing and do it well" applies here. There are already off-the-shelf caching solutions that do what you describe: for example, with Varnish you can serve cached pages immediately and update the cache contents in the background.

It would probably be better to use those than reimplement them in an uber-webapp.

Re: Javascript apps can be fully crawlable

#25

Post by wldlyinaccurate »

If you are able to "pre-render" a JavaScript app like this, then you should be serving users the pre-rendered version and then enhancing it with JavaScript after onload.

JavaScript-only apps are a blight on the web. All it takes is a bad SSL cert, or your CDN going down, and your pages become useless to the end-user.

Re: Javascript apps can be fully crawlable

#26

Post by benaiah »

dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

This is a great point. It might seem extreme, but I would advocate never using the User-Agent string to make decisions about what to serve a client. There is too much hackery and history that clouds up the User-Agent (such as every browser identifying itself as Mozilla), and it's almost always a proxy for something else that you actually want to test for.

In some rare situations, it's unavoidable, but even then I'd urge trying to rearchitect the solution to avoid it.

Re: Javascript apps can be fully crawlable

#27

Post by robmcm »

timr wrote: post #21

Don't do this. Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head. You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since…

Google actually recommend you do this, provided it's the same content that is shown with JS enabled:

https://support.google.com/webmasters/answer/66353

"JavaScript: Place the same content from the JavaScript in a tag. If you use this method, ensure the contents are exactly the same as what’s contained in the JavaScript, and that this content is shown to visitors who do not have JavaScript enabled in their browser."

This was done for years with Flash sites and I never saw Google black list anyone doing it legitimately.

You can also provide different content if you want the content to be behind a pay wall, although personally I find this is a little annoying.

Re: Javascript apps can be fully crawlable

#28

Post by benaiah »

timr wrote: post #21

Don't do this. Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head. You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since…

Hiding keyword spam behind JS doesn't make any sense in this situation - the whole point is that the JS isn't being served to Google. That's who keyword spammers are trying to fool, not actual humans.

Re: Javascript apps can be fully crawlable

#29

Post by benaiah »

dsl wrote: post #12
michaelbuckbee wrote: post #6

Earlier quoted context omitted.

Google doesn't like it when they are shown different content than a browsing user. This is roughly the equivalent of pointing Google Agent to a copy of the page requested that happens to be in Memcached instead of spinning up the full app stack to do the render.

> Google doesn't like it when they are shown different content than a browsing user. This is exactly correct. Regardless of your motivations.

Not a technically different page, specifically different content. Serving different pages to Google is fine, as long as they contain the same primary content that the real pages do. That's the whole point - so you can serve prerendered pages to Google but still have a JS-based frontend for the actual users.

Re: Javascript apps can be fully crawlable

#30

Post by [deleted] »

_lex wrote: post #8

This will get you penalized for having a website that takes forever to load. This is what happens: Googlebot requests page -> your webapp detects googlebot -> you call remote service and request that they crawl your website -> they request the page from you -> you return the regular page, with js that modifies it's look and feel -> the remote service returns the final html and css to your webapp -> your webapp return…

[deleted]
Post reply on HN