Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

1–10 of 112 posts

Re: Javascript apps can be fully crawlable

#2
I have been looking for something like this for a long time. Seems very straight forward.

I have not tested it yet, but I wonder if the speed of render will penalize you in the google results. Seems like a separate machine with a good CPU might be worthwhile if you are going to run this.

Re: Javascript apps can be fully crawlable

#6

Wouldn't the pre-render based on useragent be penalized because Google doesn't like being shown pages differently than non-Googlebot useragents?

Google doesn't like it when they are shown different content than a browsing user. This is roughly the equivalent of pointing Google Agent to a copy of the page requested that happens to be in Memcached instead of spinning up the full app stack to do the render.

Re: Javascript apps can be fully crawlable

#8
This will get you penalized for having a website that takes forever to load. This is what happens:

Googlebot requests page -> your webapp detects googlebot -> you call remote service and request that they crawl your website -> they request the page from you -> you return the regular page, with js that modifies it's look and feel -> the remote service returns the final html and css to your webapp -> your webapp returns the final html and css to Googlebot. That's gonna be just murder on your loadtimes.

If this must be done, for static pages, it should be done by grunt during build time, not by a remote service. For dynamic content, it's best to do the phantomjs rendering locally, and on an hourly (or so) schedule, since it doesn't really matter if googlebot has the latest version of your content.

Or perhaps I'm mistaken and the node-module actually calls the service hourly or so and caches results on app so it doesn't actually call the service during googlebot crawls. If that's the case, I take back my objections, but I'd recommend updating the website to say as much.

Re: Javascript apps can be fully crawlable

#9
I was under the impression that Googlebot already executes javascript on pages.

A more interesting idea would be if you do this for every user - prerender the page and send them the result, so they don't have to do the first, heavy js execution themselves. I know it sounds a bit retarded at first - you're basically using javascript as a server-side page renderer, but think about this: You can choose to prerender or not to prerender based on user agent string -- do it for people on mobile phones, but not for desktop users. You can write your entire site with just client-side page generation with javascript and let it run client-side at first, then switch to server-side prerendering once you have better hardware.

Re: Javascript apps can be fully crawlable

#10

Wouldn't the pre-render based on useragent be penalized because Google doesn't like being shown pages differently than non-Googlebot useragents?

Google doesn't like it when they are shown different content than a browsing user. This is roughly the equivalent of pointing Google Agent to a copy of the page requested that happens to be in Memcached instead of spinning up the full app stack to do the render.

[deleted]
Post reply on HN