Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Googlebot's recent improvements might revolutionize web development

blog.workhere.io

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Googlebot's recent improvements might revolutionize web development

#81

Post by tragic »

qyv wrote: post #33
jacquesm wrote: post #7

One potential problem here is that google will use this to widen the gap between it and the 'one page apps' web and other search engines (such as duckduckgo) that can't match it in resources. How strong of an advantage that will be in the long run is uncertain, I would rather see a web that ships pages with actual content in them than empty containers for a variety of reasons (most of which have to do with accessibil…

I don't see this as a problem at all, here is why: Google doesn't really have competitors in search, they just don't. I mean look at how we even define 'searching the internet' in our spoken language: 'google it'. Google has become the identity of search on the web in most people's minds. And to top it off, they are really, really good at it. Google getting better is going to widen the gap between them and everyone e…

> Nobody talks about duckduckgo because they're search results are better than google's, they talk about them because duckduckgo is all about your privacy.

'Nobody' is such a strong word.

http://devblog.avdi.org/2014/02/16/why-duckduckgo-is-better-...

Re: Googlebot's recent improvements might revolutionize web development

#82

Post by sergiosgc »

MichaelGG wrote: post #37
jacquesm wrote: post #7

One potential problem here is that google will use this to widen the gap between it and the 'one page apps' web and other search engines (such as duckduckgo) that can't match it in resources. How strong of an advantage that will be in the long run is uncertain, I would rather see a web that ships pages with actual content in them than empty containers for a variety of reasons (most of which have to do with accessibil…

>I'm fairly sure this is not the web that Tim Berners-Lee envisioned. So? None of these are convincing arguments for application developers. Having to rewrite an application to perform all its UI logic on the server side in addition to client side is a lot of work, for almost no benefit to the people paying to make the application. As a user, so long URLs work so I can send locations to other people, then most of my…

The problem is not about some God like commandment. It is about the original design of the Web, which we all believe was the reason for its success.

When you receive a GET request for an URL, and the browser tells you it accepts text/html, it is expected that you answer with the content stored at that URL in the format requested. It is not expected that you answer with an application that when run will eventually produce the content.

The correct way to do what this post is saying is to create a new mime type for this content delivery method. Then, if the browser actively tells you it accepts that content type, deliver it.

What the OP proposes is not text/html. It's something else.

Re: Googlebot's recent improvements might revolutionize web development

#83

Post by blauwbilgorgel »

Create web applications, rank as a web application.

Create web pages, rank as a web page.

This is a band-aid by Google. Developers created inaccessible websites (JS-only, no HTML fallback) and Google still wanted to give those sites a chance to be in the index. Like when Google made it possible to index text inside .swf movies. This did not mean that flash sites suddenly ranked alongside accessible websites. No, it only meant that you could now find content with a very targeted search query.

Don't think you are gaining any SEO-benefit from one-page JS-only applications, just because Google made it possible for you to start ranking.

And don't forget your responsibility as a web developer to create accessible content. Forgetting progressive enhancement, fallbacks, a noscript explanation for why you need JS, ARIA is devolution. If Google can index your site, but a blind user has a problem with your bouncy Ajax widget, then you failed catering to all your users. If you lazily let Google repair your mistakes, then soon you will be a Google-only website.

Re: Googlebot's recent improvements might revolutionize web development

#84

Post by tragic »

Like many, I am suspicious of the rather overbearing claims made on behalf of the SPA architecture.

I just launched a website. It's a weekly periodical with political analysis, word-count on articles 1500-6000. It needs to carve up the content in a few different ways (categories, issue numbers etc), decorate an article with links to other relevant content, and provide a nice CMS for non-tech people to use. So it's on Django, with the regulation sprinkling of JQuery. (If it were only techies updating it, you could probably do it with a static site generator...)

To me, the idea that you'd try and force something that is plainly a big collection of pages into a 'single page' is just philosophically bizarre, like printing Moby Dick on a square mile of paper, using some amazing origami skills to present it to the reader, all in order to save a bit of effort at the paper mill.

The googlebot business is one aspect of a bigger issue, which is that a website needs to be consumable by a host of different clients. I don't see how you can do the SPA thing without making major assumptions about those clients.

Sometimes, of course, those assumptions can be justified - it depends on the job. And Angular etc are enormously fun to play with, and handled well can enable a great UX for certain jobs. But I don't think it's 'the future'. It's another tool in the box.

Relevant here, a nice talk by John Alsopp from Full Frontal 2012:

https://www.youtube.com/watch?v=KTqIGKmCqd0

EDIT: clarification

Re: Googlebot's recent improvements might revolutionize web development

#85

Post by CMCDragonkai »

Touche wrote: post #49
iLoch wrote: post #19

Earlier quoted context omitted.

It's not difficult to set up middleware that'll render the page for any clients that require it. (For instance, we can assume any client that identifies as "bot" that's not Google probably wants a pre-rendered page, which we can do quite effortlessly. Here's one implementation for Nodejs: https://prerender.io , or you can always roll your own with something like Phantom.js.

Note that sending a different response to googlebot than what you send to normal users is a violation of Google's guidelines and can get your site penalized. Use at your own peril.

Not necessarily. This concept of HTML snapshots is actually suggested by Google as a solution.

See this: https://developers.google.com/webmasters/ajax-crawling/docs/...

We comply with Google's guidelines thing (https://snapsearch.io/documentation)

Re: Googlebot's recent improvements might revolutionize web development

#86

Post by CMCDragonkai »

BorisMelnik wrote: post #29
iLoch wrote: post #19

Earlier quoted context omitted.

It's not difficult to set up middleware that'll render the page for any clients that require it. (For instance, we can assume any client that identifies as "bot" that's not Google probably wants a pre-rendered page, which we can do quite effortlessly. Here's one implementation for Nodejs: https://prerender.io , or you can always roll your own with something like Phantom.js.

wow this is amazing. would love to see an offshoot of this where it could render a sitemap, or even keep a live sitemap up to date via cron.d or something (just hoping out loud)

Dynamic constructions of sitemaps is surprisingly difficult. We would need to poll every single page you have, just to check if you potentially have a new link to a new page in your site. And everytime you add a new page, that's a whole another page to scrape and analyse.

Re: Googlebot's recent improvements might revolutionize web development

#87

Post by workhere-io »

blauwbilgorgel wrote: post #83

Create web applications, rank as a web application. Create web pages, rank as a web page. This is a band-aid by Google. Developers created inaccessible websites (JS-only, no HTML fallback) and Google still wanted to give those sites a chance to be in the index. Like when Google made it possible to index text inside .swf movies. This did not mean that flash sites suddenly ranked alongside accessible websites. No, it o…

Don't think you are gaining any SEO-benefit from one-page JS-only applications, just because Google made it possible for you to start ranking.

No one is expecting to get any SEO benefits that "normal" pages don't have. We are expecting to get the same chance of ranking as normal pages.

You mentioned that single page apps might rank differently or worse than normal pages. Do you have any source for that? (A source that is current, since Googlebot's improvements are quite new).

Re: Googlebot's recent improvements might revolutionize web development

#88

Post by mixonic »

blauwbilgorgel wrote: post #83

Create web applications, rank as a web application. Create web pages, rank as a web page. This is a band-aid by Google. Developers created inaccessible websites (JS-only, no HTML fallback) and Google still wanted to give those sites a chance to be in the index. Like when Google made it possible to index text inside .swf movies. This did not mean that flash sites suddenly ranked alongside accessible websites. No, it o…

100% FUD.

There is no evidence that Google is going to punish my website for being rendered with JavaScript, as you imply with your first two comments.

Google is indexing the HTML generated by JavaScript, and the links in that HTML. Not some non-web custom format like SWF.

JavaScript driven sites work just fine with modern screen-readers. https://developer.mozilla.org/en-US/docs/Web/Accessibility/A... and in 2014 97.6% of screen-readers ran JavaScript http://webaim.org/projects/screenreadersurvey5/#javascript

In 2013, 92 out of 93 visitors to a UK government webpage supported JavaScript: https://gds.blog.gov.uk/2013/10/21/how-many-people-are-missi... And mixed into that 1.1% were users getting broken JS, behind firewalls, disabling JS, etc.

Google making this change does not force you to build a JavaScript-driven website, but it does make it more attractive .

Re: Googlebot's recent improvements might revolutionize web development

#89

Post by blauwbilgorgel »

workhere-io wrote: post #87
blauwbilgorgel wrote: post #83

Create web applications, rank as a web application. Create web pages, rank as a web page. This is a band-aid by Google. Developers created inaccessible websites (JS-only, no HTML fallback) and Google still wanted to give those sites a chance to be in the index. Like when Google made it possible to index text inside .swf movies. This did not mean that flash sites suddenly ranked alongside accessible websites. No, it o…

Don't think you are gaining any SEO-benefit from one-page JS-only applications, just because Google made it possible for you to start ranking. No one is expecting to get any SEO benefits that "normal" pages don't have. We are expecting to get the same chance of ranking as normal pages. You mentioned that single page apps might rank differently or worse than normal pages. Do you have any source for that? (A source tha…

>We are expecting to get the same chance of ranking as normal pages.

Then you should probably adjust this expectation. You say in your article:

>While having this sort of HTML fallback was technically possible, it added a lot of extra work to public-facing single page apps, to the point where many developers dropped the idea...

A JS-driven site with an HTML fallback is a normal page. Then you don't need any tricks or force Google to run your application and hopefully make pages out of them. Start with the fall-back and enhance.

This is a serious mistake with consequences. Tor bundle and Firefox shipped with JavaScript support, because disabling JS broke too much of the current web. It causes accessibility issues (remember when Twitter changed to hash-bang URL's?), if not for Googlebot, then for regular users (From the Webmaster Guidelines):

>Following these guidelines will help Google find, index, and rank your site.

>Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

>Make pages primarily for users, not for search engines.

I am still going on the assumption that you created a one-page application without a time-consuming fallback, and you rely on Google to make rankable pages from them. Then you leave some users standing in the cold, so why deserve to rank equal to a user-friendly accessible web page?

> ... single page apps might rank differently or worse than normal pages ...

From the original article, the most current source on this:

> Sometimes things don't go perfectly during rendering, which may negatively impact search results for your site.

> It's always a good idea to have your site degrade gracefully. This will help users enjoy your content even if their browser doesn't have compatible JavaScript implementations. It will also help visitors with JavaScript disabled or off, as well as search engines that can't execute JavaScript yet.

> Sometimes the JavaScript may be too complex or arcane for us to execute, in which case we can’t render the page fully and accurately.

> Some JavaScript removes content from the page rather than adding, which prevents us from indexing the content.

In the SEO community Googlebot's improvements were noted for a while now. See for example: http://ipullrank.com/googlebot-is-chrome/

Single page websites or application-as-content-website's are not popular among SEO's. One reason for this is that it doesn't allow for fine-grained control on keyword targeting, and keeping the site canonical, and it can waste domain authority when you have less targeted pages in the index than you can rank for. Experiment and find out for yourself.

Re: Googlebot's recent improvements might revolutionize web development

#90

Post by scotth »

ssorallen wrote: post #71
scotth wrote: post #30

Earlier quoted context omitted.

Doesn't it all depend on how long they spend on the site? If it's one page and bounce, sure, that's a terrible experience. More pages than that? Now we're starting to see savings.

If you serve a rendered page of HTML with CSS links, browsers can progressively render the page as it is downloaded. Users will notice that on the first page load, particularly on higher latency connections where round trips for resources like JavaScript files are expensive.

My point still stands. Time to load is only part of the equation.
Post reply on HN