Javascript apps can be fully crawlable • Hacker News Board

Re: Javascript apps can be fully crawlable

#71

Post by FedRegister » Mon, Oct 07, 2013, 1:25 PM UTC

eonil wrote: post #58

Static rendering of dynamic content? I don't think this does make sense. If it's pre-rednered, it's missing something. If it has all the data at first, then it's not dynamic. Pre-rendered(static) javascript app(dynamic)...? Hmm... I don't see anything more than something like JWT in JS instead of Java?

>Static rendering of dynamic content? I don't think this does make sense.

Bro do you even Web 1.0? That's what CGI scripts in Perl did! Pull the data from the database, generate HTML (no JavaScript back then!) on the fly, and send to the browser.

Re: Javascript apps can be fully crawlable

#72

Post by nailer » Mon, Oct 07, 2013, 1:25 PM UTC

philbo wrote: post #63

Isofarro wrote: post #44

An entire project written to simulate progressive enhancement (badly). One that only works for specified whitelisted User-Agents, instead of being based on capability. I'm also not understanding the use-case for this project. Everytime the topic of "Web Apps", "JavaScript Apps", "Single page web apps" comes up, evangelists point out that they are applications (or skyscrapers), not just fancy decorators for website co…

+100 Why some developers are so against progressive enhancement mystifies me. It is an elegant solution that actually works in all cases rather than an ugly hack that should probably work in the majority of cases. How can there even be a dispute about it? It's insane!

> Why some developers are so against progressive enhancement mystifies me.

There's another common adage: HTML is content. CSS is presentation. JS is behaviour.

Some public web apps simply don't work without behaviour.

Re: Javascript apps can be fully crawlable

#73

Post by thoop » Mon, Oct 07, 2013, 1:35 PM UTC

gorm wrote: post #57

Very cool, but something I don't get: - Try to go to prerender.io, press "Install It -> Ruby on rails". Now it loads the ruby on rails example. - Then go all the way down and change to "Prerendereed content". Pressing "Install It -> Ruby on rails" doesn't do anything now. Shouldn't it render the same content? "Add the middleware gem to your Gemfile..." and so on.

prerender.io uses js(bootstrap) for the tab switching. So the prerendered page doesn't do anything because it doesn't load that javascript.

Re: Javascript apps can be fully crawlable

#74

Post by MostAwesomeDude » Mon, Oct 07, 2013, 1:47 PM UTC

stuartmemo wrote: post #67

acqq wrote: post #65

I surf with JavaScript turned off and I see just a blank page. If it's "crawlable" I certainly expect it to be visible to me without turning JavaScript on.

Why? You're not a web crawler.

Then I would humbly and politely suggest that the website use a header like `Vary: User-Agent` in its response to indicate that different user agents are given different responses. Alternatively, use a 300 Multiple Choices status code to present the two different versions of the site: The crawlable version, and the JS-encumbered version.

I understand that RFC 2616 does not forbid this technique, but something is not right simply because it is not wrong, and it's deceptive in either case. Do what is right and good, not what is merely not wrong.

Re: Javascript apps can be fully crawlable

#75

Post by bceagle » Mon, Oct 07, 2013, 2:04 PM UTC

dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

> This is a great approach

No, it is not. While this will certainly help your client side app get indexed, it is not 'great'. Other commenters on this thread bring up a number of valid concerns, but in my mind it comes down to two very simply things.

One is that when you are fighting for the top spot in organic traffic, this won't cut it. Off-page SEO is more important than on-page optimizations, but it on-page optimizations still have value.

The other issue is that this approach assumes that the client side rendered view at a particular hash is exactly what should be initially rendered on the server side. While this could work in some cases, it is my experience that it either creates a weird user experience and/or you end up doing hacks on the client side in order to ensure PhantomJS captures the right html.

This is a fine solution for some use cases, but I really hope that the community doesn't think this is the future. This is a temporary hack until we get a good server/client rendering framework in place OR all search engines evolve to capture pure client side apps without any of this.

Re: Javascript apps can be fully crawlable

#76

Post by alanlewis » Mon, Oct 07, 2013, 2:12 PM UTC

Isofarro wrote: post #53

dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

That Google Ajax crawler spec is no magic bullet. Nick Denton: "Dip in uniques largely because of drop in Google refers. Pageviews (which are driven more by core audience) less affected." -- http://twitter.com/nicknotned/status/61152134929981440 Nick Denton: "Google does not fully support "hashbang" URLs. So we're eliminating them rather than waiting for Mountain View." -- http://twitter.com/nicknotned/status/6146585…

These tweets are from 2.5 years ago. Has the google bot improved since then? (Honest question)

Re: Javascript apps can be fully crawlable

#77

Post by selvakn » Mon, Oct 07, 2013, 2:15 PM UTC

Shameless plug: https://github.com/selvakn/rack_phantom

Similar idea, but implemented without a server for rendering, with a phantomjs process. And only for rails/rack app.

Re: Javascript apps can be fully crawlable

#78

Post by thoop » Mon, Oct 07, 2013, 2:16 PM UTC

dwwoelfel wrote: post #23

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

Thanks, I'll work on adding that. I'd still like to use a useragent fall-back for the crawlers that might not use the _escaped_fragment_ protocol.

Re: Javascript apps can be fully crawlable

#79

Post by welly » Mon, Oct 07, 2013, 2:17 PM UTC

acqq wrote: post #65

I surf with JavaScript turned off and I see just a blank page. If it's "crawlable" I certainly expect it to be visible to me without turning JavaScript on.

> I surf with JavaScript turned off

Why would you do this? Genuinely interested. Do you browse the web with JavaScript turned off the majority of the time or just in this particular example?

Re: Javascript apps can be fully crawlable

#80

Post by gbadman » Mon, Oct 07, 2013, 2:18 PM UTC

philbo wrote: post #63

Isofarro wrote: post #44

An entire project written to simulate progressive enhancement (badly). One that only works for specified whitelisted User-Agents, instead of being based on capability. I'm also not understanding the use-case for this project. Everytime the topic of "Web Apps", "JavaScript Apps", "Single page web apps" comes up, evangelists point out that they are applications (or skyscrapers), not just fancy decorators for website co…

+100 Why some developers are so against progressive enhancement mystifies me. It is an elegant solution that actually works in all cases rather than an ugly hack that should probably work in the majority of cases. How can there even be a dispute about it? It's insane!

For the vast majority of websites, this is true and I agree.

However, there are now websites that are more akin to applications than to the traditional website of yore and that are meaningless without Javascript. This is not a bad thing; it is just the opening up of the web platform to new opportunities.

Some of these emerging applications may want some of their content to be searchable and so I argue that the posted solution is solving a real problem and is of value to users.