Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Javascript apps can be fully crawlable

#31

Post by benaiah »

anonymous wrote: post #9

I was under the impression that Googlebot already executes javascript on pages. A more interesting idea would be if you do this for every user - prerender the page and send them the result, so they don't have to do the first, heavy js execution themselves. I know it sounds a bit retarded at first - you're basically using javascript as a server-side page renderer, but think about this: You can choose to prerender or n…

Something similar to that, albeit slightly more elegant, is the work that AirBnB has done with their rendr [0] project, which serves prerendered content that's then rerendered with JS if it needs to be changed. You can do similar things with non-Backbone stacks, of course.

[0]: https://github.com/airbnb/rendr

Re: Javascript apps can be fully crawlable

#32

Post by robmcm »

benaiah wrote: post #26
dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

This is a great point. It might seem extreme, but I would advocate never using the User-Agent string to make decisions about what to serve a client. There is too much hackery and history that clouds up the User-Agent (such as every browser identifying itself as Mozilla), and it's almost always a proxy for something else that you actually want to test for. In some rare situations, it's unavoidable, but even then I'd u…

Actually Google recomend using the IP address of the bot vai a DNS lookup:

http://googlewebmastercentral.blogspot.co.uk/2006/09/how-to-...

Re: Javascript apps can be fully crawlable

#33

Post by rurounijones »

dsl wrote: post #14
est wrote: post #11

Earlier quoted context omitted.

web apps today are so much more than ajax. You have to actually a full blown DOM tree to get what real user-agents renders

If only Google had access to a full blown browser they could use in the crawl engine...

* At scale, without massive performance drops

Re: Javascript apps can be fully crawlable

#34

Post by timr »

robmcm wrote: post #27
timr wrote: post #21

Don't do this. Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head. You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since…

Google actually recommend you do this, provided it's the same content that is shown with JS enabled: https://support.google.com/webmasters/answer/66353 "JavaScript: Place the same content from the JavaScript in a tag. If you use this method, ensure the contents are exactly the same as what’s contained in the JavaScript, and that this content is shown to visitors who do not have JavaScript enabled in their browser." T…

Placing the text content on the same page in a tag is entirely different than rendering different content based on user agent. That's what the noscript tag is meant to do -- Google is telling you to follow best practices for fallback to non-JS browsers.

Re: Javascript apps can be fully crawlable

#35

Post by richardwhiuk »

dsl wrote: post #14
est wrote: post #11

Earlier quoted context omitted.

web apps today are so much more than ajax. You have to actually a full blown DOM tree to get what real user-agents renders

If only Google had access to a full blown browser they could use in the crawl engine...

The compute costs would be extraordinarily expensive.

Re: Javascript apps can be fully crawlable

#36

Post by benaiah »

robmcm wrote: post #32
benaiah wrote: post #26

Earlier quoted context omitted.

This is a great point. It might seem extreme, but I would advocate never using the User-Agent string to make decisions about what to serve a client. There is too much hackery and history that clouds up the User-Agent (such as every browser identifying itself as Mozilla), and it's almost always a proxy for something else that you actually want to test for. In some rare situations, it's unavoidable, but even then I'd u…

Actually Google recomend using the IP address of the bot vai a DNS lookup: http://googlewebmastercentral.blogspot.co.uk/2006/09/how-to-...

That makes sense, though in an ideal world there would be an abstracted header that said "hey, I'm not gonna render JS the way a regular browser will, so send me something prerendered". Then you could write something that would actually be future-proof and work with other search engines.

The way Google suggests there actually seems a little bit nefarious, as it makes it hard-coded to Google instead of working for any search engine.

Re: Javascript apps can be fully crawlable

#37

Post by timr »

benaiah wrote: post #28
timr wrote: post #21

Don't do this. Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head. You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since…

Hiding keyword spam behind JS doesn't make any sense in this situation - the whole point is that the JS isn't being served to Google. That's who keyword spammers are trying to fool, not actual humans.

It works either way. If I'm allowed to serve a big blob of javascript to my users and not to the googlebot, I load up my googlebot page with keyword content, and use the JS blob to render my "secret" page to real users.

Re: Javascript apps can be fully crawlable

#38

Post by Volpe »

rurounijones wrote: post #33
dsl wrote: post #14

Earlier quoted context omitted.

If only Google had access to a full blown browser they could use in the crawl engine...

* At scale, without massive performance drops

I'm confused, search indexing isn't a realtime exercise... Why would performance be an issue? Running a headless browser vs running "whatever it is they run that can execute JS" doesn't seem like a huge leap...

Re: Javascript apps can be fully crawlable

#39

Post by stephenheron »

timr wrote: post #21

Don't do this. Rendering different content based on user agent is tempting the webspam gods. Rendering nothing but a big gob of javascript to non-googlebot user agents is a recipe to get the banhammer dropped on your head. You're either gambling that Google is smart enough to know that your particular big gob of javascript isn't cloaking keyword spam (in which case you should just depend on their JS evaluation, since…

Google does have a section within their guidelines on creating "HTML Snapshots". "If a lot of your content is created in JavaScript, you may want to consider using a technology such as a headless browser to create an HTML snapshot." https://developers.google.com/webmasters/ajax-crawling/docs/...

Re: Javascript apps can be fully crawlable

#40

Post by est »

Volpe wrote: post #38
rurounijones wrote: post #33

Earlier quoted context omitted.

* At scale, without massive performance drops

I'm confused, search indexing isn't a realtime exercise... Why would performance be an issue? Running a headless browser vs running "whatever it is they run that can execute JS" doesn't seem like a huge leap...

Have you ever experienced web apps that laggs like crap? Yeah think about that x 10000 million web pages.
Post reply on HN