Googlebot's recent improvements might revolutionize web development • Hacker News Board

Re: Googlebot's recent improvements might revolutionize web development

#91

Post by Touche » Wed, May 28, 2014, 12:40 PM UTC

CMCDragonkai wrote: post #85

Touche wrote: post #49

Earlier quoted context omitted.

Note that sending a different response to googlebot than what you send to normal users is a violation of Google's guidelines and can get your site penalized. Use at your own peril.

Not necessarily. This concept of HTML snapshots is actually suggested by Google as a solution. See this: https://developers.google.com/webmasters/ajax-crawling/docs/... We comply with Google's guidelines thing ( https://snapsearch.io/documentation )

No where in that article does it say it's ok to only serve the snapshot to googlebot. Serving different content to googlebot than what you serve to users is called cloaking and is against their guidelines: https://support.google.com/webmasters/answer/66355

I've invested a significant amount of time in this topic and would love if you were right, but I've never seen the money quote that it's ok to do this. In fact, everything I've read says that you have to treat search bots the same as you treat normal users.

Re: Googlebot's recent improvements might revolutionize web development

#92

Post by blauwbilgorgel » Wed, May 28, 2014, 12:57 PM UTC

mixonic wrote: post #88

blauwbilgorgel wrote: post #83

Create web applications, rank as a web application. Create web pages, rank as a web page. This is a band-aid by Google. Developers created inaccessible websites (JS-only, no HTML fallback) and Google still wanted to give those sites a chance to be in the index. Like when Google made it possible to index text inside .swf movies. This did not mean that flash sites suddenly ranked alongside accessible websites. No, it o…

100% FUD. There is no evidence that Google is going to punish my website for being rendered with JavaScript, as you imply with your first two comments. Google is indexing the HTML generated by JavaScript, and the links in that HTML. Not some non-web custom format like SWF. JavaScript driven sites work just fine with modern screen-readers. https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... and in 2014 97.…

If I wanted to imply that Google will punish your website for being rendered with JavaScript, I probably would have said so. It would likely be false too, as it is less of a punishment, than it is not maximizing your chance to rank (to put your best foot forward as a website).

Accessibility is not a numbers game. In many countries it is a legal requirement. And adhering to the WCAG means providing non-JS fallbacks or progressive enhancement. RMS not being able to access your content is an accessibility issue too, it does not have to involve a disability. It can be technical in nature, like disabling JS or being behind a corporate firewall, or your browser not supporting pushstate.

If you want to look at stats, take a look at the stats and surveys on accessibility of dynamic web applications. Just because your screenreader supports JavaScript does not mean you have no accessibility issues due to JavaScript. Rich internet applications should use WAI-ARIA. I don't think people who create websites without a fallback (avoiding this issue entirely), will worry about creating websites with ARIA-support. And if they do care about such accessibility, they should also provide a non-ARIA non-JS fallback.

Google making this change makes it possible to have your non-fallback JS-only application be indexed. It does not make it more attractive from an SEO or accessibility viewpoint.

Re: Googlebot's recent improvements might revolutionize web development

#93

Post by CMCDragonkai » Wed, May 28, 2014, 1:57 PM UTC

Touche wrote: post #91

CMCDragonkai wrote: post #85

Earlier quoted context omitted.

Not necessarily. This concept of HTML snapshots is actually suggested by Google as a solution. See this: https://developers.google.com/webmasters/ajax-crawling/docs/... We comply with Google's guidelines thing ( https://snapsearch.io/documentation )

No where in that article does it say it's ok to only serve the snapshot to googlebot. Serving different content to googlebot than what you serve to users is called cloaking and is against their guidelines: https://support.google.com/webmasters/answer/66355 I've invested a significant amount of time in this topic and would love if you were right, but I've never seen the money quote that it's ok to do this. In fact, ev…

The title of the article says "How do I create an HTML snapshot?"

The FAQ (https://developers.google.com/webmasters/ajax-crawling/docs/...) goes into more detail regarding the concept of _escaped_fragment_ which is used in so that your server can respond with a static snapshot instead of javascript.

Look at the diagram at the bottom of this page (https://developers.google.com/webmasters/ajax-crawling/docs/...)

It explicitly says "snapshots"

Re: Googlebot's recent improvements might revolutionize web development

#94

Post by mixonic » Wed, May 28, 2014, 2:48 PM UTC

blauwbilgorgel wrote: post #92

mixonic wrote: post #88

Earlier quoted context omitted.

100% FUD. There is no evidence that Google is going to punish my website for being rendered with JavaScript, as you imply with your first two comments. Google is indexing the HTML generated by JavaScript, and the links in that HTML. Not some non-web custom format like SWF. JavaScript driven sites work just fine with modern screen-readers. https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... and in 2014 97.…

If I wanted to imply that Google will punish your website for being rendered with JavaScript, I probably would have said so. It would likely be false too, as it is less of a punishment, than it is not maximizing your chance to rank (to put your best foot forward as a website). Accessibility is not a numbers game. In many countries it is a legal requirement. And adhering to the WCAG means providing non-JS fallbacks or…

Web accessibility, as we commonly use the term, pertains to creating a website that disabled users can interact with an navigate. It does not pertain to those who choose to or are forced to disable JavaScript (the RMS example). Creating an accessible site is a challenge regardless of what technologies you pick, for sure. Just as saying "just because your screenreader supports JavaScript does not mean you have no accessibility issues", just because your website uses JavaScript doesn't mean you have accessibility issues. A plain HTML website can have accessibility issues. So can a JavaScript one.

AFAIK, nothing in WCAG says you must have a non-JavaScript fallback to adhere to their standard. If you can back that up I am all ears, I would be interested to read it.

> Google making this change makes it possible to have your non-fallback JS-only application be indexed. It does not make it more attractive from an SEO or accessibility viewpoint.

The attractiveness of JS heavy development is not in an inherent SEO or accessibility benefit. Absolutely true.

The benefit is a development style that is more productive, giving me more time as a developer to focus on solving the problem at hand, be it business logic, SEO, or accessibility. You can debate this benefit, but don't imply that single-page apps cannot have SEO on par with HTML sites and good accessibility.

Re: Googlebot's recent improvements might revolutionize web development

#95

Post by allendoerfer » Wed, May 28, 2014, 4:21 PM UTC

This news article seems to come up every few years. Nevertheless, the niche of apps, which can profit of this, is quite small.

Either you have a highly interaction-heavy web-app, where it makes sense to execute most of the code on the client and deliver the content as JSON or you have a content-heavy website, where it makes sense to deliver cached content to the client.

There are some apps in between, which are highly interactive and content heavy like web versions of social apps. For them additionally the question arises, if they want to be crawled by Google or Google wants to index their content.

To profit from this, you need an app, which content the users search for and interact with several times after they have found it. So i guess, "revolutionize" seems a bit much to me.

Re: Googlebot's recent improvements might revolutionize web development

#96

Post by blauwbilgorgel » Thu, May 29, 2014, 3:32 PM UTC

mixonic wrote: post #94

blauwbilgorgel wrote: post #92

Earlier quoted context omitted.

If I wanted to imply that Google will punish your website for being rendered with JavaScript, I probably would have said so. It would likely be false too, as it is less of a punishment, than it is not maximizing your chance to rank (to put your best foot forward as a website). Accessibility is not a numbers game. In many countries it is a legal requirement. And adhering to the WCAG means providing non-JS fallbacks or…

Web accessibility, as we commonly use the term, pertains to creating a website that disabled users can interact with an navigate. It does not pertain to those who choose to or are forced to disable JavaScript (the RMS example). Creating an accessible site is a challenge regardless of what technologies you pick, for sure. Just as saying "just because your screenreader supports JavaScript does not mean you have no acce…

>Web accessibility, as we commonly use the term, pertains to creating a website that disabled users can interact with an navigate. It does not pertain to those who choose to or are forced to disable JavaScript...

Often web accessibility focuses on people with a disability, correct. Accessibility, like I said, really is more than that, though. From the Wiki: Accessibility is the degree to which a product, device, service, or environment is available to as many people as possible. Hence it does pertain to those who choose to or are forced to disable JavaScript. It literally means _as many people as possible_, RMS included. Even the WCAG do not solely focus on assistive technologies, but include "a wide variety of user agents".

The comment "just because your screenreader supports JavaScript does not mean you have no accessibility issues" was in reply to your statistics on JS-support for screenreaders. 98% of screenreaders supporting JavaScript is moot when less than 75% of browsers support pushState. In other words: You leave much more than 2% of users incapable of accessing your content. WebAim Surveys show that people have increasingly more trouble accessing content on JS-heavy social sites and dynamic web applications.

> A plain HTML website can have accessibility issues. So can a JavaScript one.

A JavaScript site can have a problem. If you serve it without a fallback (under the assumption that 98% of your users can access it that way) then it has a problem for sure. I have nothing against JavaScript. I have a problem with JavaScript sites that don't provide a fallback or weren't build according to progressive enhancement principles.

>AFAIK, nothing in WCAG says you must have a non-JavaScript fallback to adhere to their standard.

It said so specifically in WCAG 1. WCAG 2 is more ambiguous. You can have a no-fallback application that requires JavaScript provided: You can not show the content in any other way (a fallback is impossible), and you clearly explain in why JavaScript is required.

Where a fallback IS possible, not providing one lowers accessibility. This is the relevant principle:

Principle 4: Robust - Content must be robust enough that it can be interpreted reliably by a wide variety of user agents, including assistive technologies.

If you do not provide a fall-back and require JavaScript then your content can not be interpreted reliably by a wide variety of user agents. Not providing a fall-back goes against this principle.

Relevant guideline:

Guideline 4.1 Compatible: Maximize compatibility with current and future user agents, including assistive technologies.

JS-only non-fallback sites do not maximize compatibility, they minimize it, breaking this guideline.

Government 508 guidelines for accessibility:

When possible, the functionality should be available without requiring JavaScript. When this is not possible, the functionality should fail gracefully (i.e., inform the user that JavaScript is required).

Webaccessibility.com best practices:

Ensure elements that use ARIA provide non-ARIA fallback accessible content.

Since you should markup your rich web apps with ARIA, and you should provide a non-ARIA accessible fallback, you should provide an accessible fallback for your rich web app.

I do know that this can be a point of debate, and that is fine. It is up for interpretation what "maximize compatibility" means to you. If you have legal obligations to maximize compatibility (like government organizations in The Netherlands) then this becomes a harder rule.

> The benefit is a development style that is more productive, giving me more time as a developer to focus on solving the problem at hand, be it business logic, SEO, or accessibility.

I really don't understand this way of thinking. If you want to spend time on accessibility, start with a fallback, don't create a website without a fallback and then cheer on the idea that now you have time left to fix the problem you created a few minutes before that...

If you want to solve problems with SEO, don't start out by creating one :D

> but don't imply that single-page apps cannot have SEO on par with HTML sites and good accessibility.

Good accessibility means good SEO. No fallback means poor accessibility. Draw your own conclusions (Socrates is mortal?).

Re: Googlebot's recent improvements might revolutionize web development

#97

Post by gildas » Thu, May 29, 2014, 4:45 PM UTC

BorisMelnik wrote: post #29

iLoch wrote: post #19

Earlier quoted context omitted.

It's not difficult to set up middleware that'll render the page for any clients that require it. (For instance, we can assume any client that identifies as "bot" that's not Google probably wants a pre-rendered page, which we can do quite effortlessly. Here's one implementation for Nodejs: https://prerender.io , or you can always roll your own with something like Phantom.js.

wow this is amazing. would love to see an offshoot of this where it could render a sitemap, or even keep a live sitemap up to date via cron.d or something (just hoping out loud)

You can use SEO4Ajax [1] to crawl your SPA and generate an up to date sitemap dynamically.

[1] http://www.seo4ajax.com/