Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Javascript apps can be fully crawlable

#81

Post by thoop »

bceagle wrote: post #75
dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

> This is a great approach No, it is not. While this will certainly help your client side app get indexed, it is not 'great'. Other commenters on this thread bring up a number of valid concerns, but in my mind it comes down to two very simply things. One is that when you are fighting for the top spot in organic traffic, this won't cut it. Off-page SEO is more important than on-page optimizations, but it on-page optim…

There definitely are valid concerns with this approach...but until the search engines pick up the slack, this is the solution that we're left with. It's definitely not ideal, but I prefer it over having non-DRY code just to serve incomplete HTML to crawlers.

Re: Javascript apps can be fully crawlable

#82

Post by bceagle »

thoop wrote: post #81
bceagle wrote: post #75

Earlier quoted context omitted.

> This is a great approach No, it is not. While this will certainly help your client side app get indexed, it is not 'great'. Other commenters on this thread bring up a number of valid concerns, but in my mind it comes down to two very simply things. One is that when you are fighting for the top spot in organic traffic, this won't cut it. Off-page SEO is more important than on-page optimizations, but it on-page optim…

There definitely are valid concerns with this approach...but until the search engines pick up the slack, this is the solution that we're left with. It's definitely not ideal, but I prefer it over having non-DRY code just to serve incomplete HTML to crawlers.

The alternative solution is pushstate with an acceptance that you will need to either do some duplication of effort on the server side rendering or creating something along the lines of the AirBnb Rendr framework. I am in the process of doing the former with the plan to do the latter with a future iteration.

Re: Javascript apps can be fully crawlable

#83

Post by Isofarro »

nailer wrote: post #72
philbo wrote: post #63

Earlier quoted context omitted.

+100 Why some developers are so against progressive enhancement mystifies me. It is an elegant solution that actually works in all cases rather than an ugly hack that should probably work in the majority of cases. How can there even be a dispute about it? It's insane!

> Why some developers are so against progressive enhancement mystifies me. There's another common adage: HTML is content. CSS is presentation. JS is behaviour. Some public web apps simply don't work without behaviour.

> Some public web apps simply don't work without behaviour.

Every app that uses a solution like this to generate static views of a website is an app that simply works without behaviour.

Re: Javascript apps can be fully crawlable

#84

Post by raynjamin »

Isofarro wrote: post #44

An entire project written to simulate progressive enhancement (badly). One that only works for specified whitelisted User-Agents, instead of being based on capability. I'm also not understanding the use-case for this project. Everytime the topic of "Web Apps", "JavaScript Apps", "Single page web apps" comes up, evangelists point out that they are applications (or skyscrapers), not just fancy decorators for website co…

What would you do if you required SEO enhancement AND dynamic loading of content? Are you supposed to just let that portion of the site go without indexing? Surely there are sites that have both requirements.

What's the alternative?

Re: Javascript apps can be fully crawlable

#85

Post by acqq »

welly wrote: post #79
acqq wrote: post #65

I surf with JavaScript turned off and I see just a blank page. If it's "crawlable" I certainly expect it to be visible to me without turning JavaScript on.

> I surf with JavaScript turned off Why would you do this? Genuinely interested. Do you browse the web with JavaScript turned off the majority of the time or just in this particular example?

I keep JavaScript turned off by default. Then I turn it on for only a few sites of critical importance for me which would not function otherwise. And I don't feel I miss anything, most of the content I care about is still HTML and it should remain so. JavaScript is not needed to show me the text.

That way the chances for cross site sripting attacks are greatly reduced and the content appears much faster.

Re: Javascript apps can be fully crawlable

#86

Post by robmcm »

mk3 wrote: post #41
robmcm wrote: post #32

Earlier quoted context omitted.

Actually Google recomend using the IP address of the bot vai a DNS lookup: http://googlewebmastercentral.blogspot.co.uk/2006/09/how-to-...

You do realize that you gave a link which is from 2006? And more recent recommendations does not include that. [EDIT] OK as I was downvoted I will clarify my point: https://developers.google.com/webmasters/ajax-crawling/docs/... This is recommended practice for crawling javascript generated pages, no need to lookup for spiders IP address as someone mentioned.

True

Re: Javascript apps can be fully crawlable

#87

Post by abraham »

Isofarro wrote: post #53
dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

That Google Ajax crawler spec is no magic bullet. Nick Denton: "Dip in uniques largely because of drop in Google refers. Pageviews (which are driven more by core audience) less affected." -- http://twitter.com/nicknotned/status/61152134929981440 Nick Denton: "Google does not fully support "hashbang" URLs. So we're eliminating them rather than waiting for Mountain View." -- http://twitter.com/nicknotned/status/6146585…

You can use the meta tags with normal URLs and completely ignore hashbang urls.

Re: Javascript apps can be fully crawlable

#88

Post by Isofarro »

alanlewis wrote: post #76
Isofarro wrote: post #53

Earlier quoted context omitted.

That Google Ajax crawler spec is no magic bullet. Nick Denton: "Dip in uniques largely because of drop in Google refers. Pageviews (which are driven more by core audience) less affected." -- http://twitter.com/nicknotned/status/61152134929981440 Nick Denton: "Google does not fully support "hashbang" URLs. So we're eliminating them rather than waiting for Mountain View." -- http://twitter.com/nicknotned/status/6146585…

These tweets are from 2.5 years ago. Has the google bot improved since then? (Honest question)

Considering the intention behind the Google document is to enable support for existing Ajax applications, and not the cornerstone of crawlability of newly built apps, probably not.

Also, the same document that's quoted in defence of these Web (unfriendly) Apps is https://developers.google.com/webmasters/ajax-crawling/

Where in the first section of that document: https://developers.google.com/webmasters/ajax-crawling/docs/... There is this:

"If you're starting from scratch, one good approach is to build your site's structure and navigation using only HTML. Then, once you have the site's pages, links, and content in place, you can spice up the appearance and interface with AJAX. Googlebot will be happy looking at the HTML, while users with modern browsers can enjoy your AJAX bonuses."

Re: Javascript apps can be fully crawlable

#89

Post by Isofarro »

raynjamin wrote: post #84
Isofarro wrote: post #44

An entire project written to simulate progressive enhancement (badly). One that only works for specified whitelisted User-Agents, instead of being based on capability. I'm also not understanding the use-case for this project. Everytime the topic of "Web Apps", "JavaScript Apps", "Single page web apps" comes up, evangelists point out that they are applications (or skyscrapers), not just fancy decorators for website co…

What would you do if you required SEO enhancement AND dynamic loading of content? Are you supposed to just let that portion of the site go without indexing? Surely there are sites that have both requirements. What's the alternative?

Progressive enhancement. It is a web development best practice.

You will find that "dynamic loading of content" doesn't automatically mean "no content served by HTML under any circumstances". This is an error perpetuated by these JavaScript-only frameworks.

For example, bustle.com - there is absolutely no customer experience reason for the website not to have the content loaded at the HTML layer and then progressively enhanced with the customer experience additions. The content here isn't tied exclusively to the behaviour layer.

Re: Javascript apps can be fully crawlable

#90

Post by raynjamin »

Isofarro wrote: post #89
raynjamin wrote: post #84

Earlier quoted context omitted.

What would you do if you required SEO enhancement AND dynamic loading of content? Are you supposed to just let that portion of the site go without indexing? Surely there are sites that have both requirements. What's the alternative?

Progressive enhancement. It is a web development best practice. You will find that "dynamic loading of content" doesn't automatically mean "no content served by HTML under any circumstances". This is an error perpetuated by these JavaScript-only frameworks. For example, bustle.com - there is absolutely no customer experience reason for the website not to have the content loaded at the HTML layer and then progressivel…

"The content here isn't tied exclusively to the behaviour layer."

Can you elaborate on a situation where that the content and the behaviour are tied together, and what you would do in that case?

From my understanding, Facebook's BigPipe loads content in modules to reduce user perceived latency. If I'm building X site and wanted that same behavior (since I've heard on several occasions that there is a direct correlation between page load times and user engagement), is my only option to sacrifice SEO?

Post reply on HN