Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Googlebot's recent improvements might revolutionize web development

blog.workhere.io

Post reply on HN
Print view View original

11–20 of 97 posts

Re: Googlebot's recent improvements might revolutionize web development

#11

Post by epistasis »

jacquesm wrote: post #7

One potential problem here is that google will use this to widen the gap between it and the 'one page apps' web and other search engines (such as duckduckgo) that can't match it in resources. How strong of an advantage that will be in the long run is uncertain, I would rather see a web that ships pages with actual content in them than empty containers for a variety of reasons (most of which have to do with accessibil…

I agree 100%, and see other problems for users as well. The worst web pages I encounter are the javascript-constructed DOM pages. And by far the worst offender is the odious Google Web Toolkit.

It may be that I encounter some of these "modern" pages without knowing it because some dev has put in the time to make it work well, it seems that the vast majority are absolutely terrible. It's a small decrease in developer effort for a huge decrease in user satisfaction. I remember the terrible terrible #! days of Twitter with sadness.

Some developers have a tendency to go for the internally sophisticated/beautiful in preference to the best experience for the user. I hope that blog posts like this one don't let loose these developers' worst tendencies.

Re: Googlebot's recent improvements might revolutionize web development

#12

Post by nkuttler »

jacquesm wrote: post #7

One potential problem here is that google will use this to widen the gap between it and the 'one page apps' web and other search engines (such as duckduckgo) that can't match it in resources. How strong of an advantage that will be in the long run is uncertain, I would rather see a web that ships pages with actual content in them than empty containers for a variety of reasons (most of which have to do with accessibil…

One page apps that aren't crawlable don't want to be crawled, don't make the necessary work, or are simply incompetent. Making an ajax site crawlable isn't exactly rocket science.

The gap this will really widen is the one between sites that do the necessary work themselves, and those who don't.

Re: Googlebot's recent improvements might revolutionize web development

#13

Post by fixermark »

Yetanfou wrote: post #9
justinph wrote: post #4

We're still using hypertext transfer protocol, right? You need to send hypertext down the wire. We shouldn't let one company, google, dictate how the web works, simply because of their proprietary technological innovation.

Google is not dictating anything so there is no need to vent your dislike, of, Google, in that way. If anything, this releases developers from a constraint on how sites are built. That said, I'd rather see more server-side HTML instead of client-side JS when it comes to the web. If you're developing a game, client-side JS is fine. If you're serving textual content with the odd image, please serve it in the way the we…

If my product is well-formatted data, why should I build a server that needs to know how to vend that data in a machine-readable visual-formatting-agnostic format (such as JSON or XML) and also a targeted-for-human-consumption format such as HTML? It's a reasonable architectural decision to build one server that knows only how to vend JSON and an associated viewer (that happens to use HTML and JavaScript for presentation purposes) that knows how to consume and render that JSON.

Re: Googlebot's recent improvements might revolutionize web development

#14

Post by pessimizer »

fixermark wrote: post #13
Yetanfou wrote: post #9

Earlier quoted context omitted.

Google is not dictating anything so there is no need to vent your dislike, of, Google, in that way. If anything, this releases developers from a constraint on how sites are built. That said, I'd rather see more server-side HTML instead of client-side JS when it comes to the web. If you're developing a game, client-side JS is fine. If you're serving textual content with the odd image, please serve it in the way the we…

If my product is well-formatted data, why should I build a server that needs to know how to vend that data in a machine-readable visual-formatting-agnostic format (such as JSON or XML) and also a targeted-for-human-consumption format such as HTML? It's a reasonable architectural decision to build one server that knows only how to vend JSON and an associated viewer (that happens to use HTML and JavaScript for presenta…

Because it pushes work onto the client, and the client has an indeterminate number of applications competing for the same indeterminate speed and size of CPU and RAM. You know exactly what your situation is on the server, and you can decouple data and presentation without putting them on different sides of the wire.

Re: Googlebot's recent improvements might revolutionize web development

#15

Post by ksk »

pessimizer wrote: post #14
fixermark wrote: post #13

Earlier quoted context omitted.

If my product is well-formatted data, why should I build a server that needs to know how to vend that data in a machine-readable visual-formatting-agnostic format (such as JSON or XML) and also a targeted-for-human-consumption format such as HTML? It's a reasonable architectural decision to build one server that knows only how to vend JSON and an associated viewer (that happens to use HTML and JavaScript for presenta…

Because it pushes work onto the client, and the client has an indeterminate number of applications competing for the same indeterminate speed and size of CPU and RAM. You know exactly what your situation is on the server, and you can decouple data and presentation without putting them on different sides of the wire.

Well said. Too many websites these days are JS heavy.

Re: Googlebot's recent improvements might revolutionize web development

#16

Post by fixermark »

pessimizer wrote: post #14
fixermark wrote: post #13

Earlier quoted context omitted.

If my product is well-formatted data, why should I build a server that needs to know how to vend that data in a machine-readable visual-formatting-agnostic format (such as JSON or XML) and also a targeted-for-human-consumption format such as HTML? It's a reasonable architectural decision to build one server that knows only how to vend JSON and an associated viewer (that happens to use HTML and JavaScript for presenta…

Because it pushes work onto the client, and the client has an indeterminate number of applications competing for the same indeterminate speed and size of CPU and RAM. You know exactly what your situation is on the server, and you can decouple data and presentation without putting them on different sides of the wire.

The counter-point is that (a) the number of applications competing for resources is semi-determinate, constrained by the user's patience and willingness to upgrade and (b) the number of users the server must support is similarly semi-determinant, constrained by the popularity of the service, size of data a user could want to manipulate, and the desired speed of those manipulations. There's an argument to be made for the economy of scale of pushing some of the presentation work to the side of the wire where there is a user who wants to consume the data; I wouldn't want to have to use, say, Google Spreadsheets as a server-side-only product.

Note that we can reduce this line of thinking to absurdity if we substitute JavaScript with, say, video; some clients are incapable of rendering video, some clients are constrained in the size of video they can render. Do we therefore never stream video down the wire, and require the client to grab the whole video element wholesale as a single request-response? Of course not; users don't want to wait that long. So developers make an educated guess on what their clients' user-agents can do and set a sane bar for size and scope of video content. A similar sane bar can be set for richness of client-side executable code interactions.

Re: Googlebot's recent improvements might revolutionize web development

#17

Post by justinph »

o_____________o wrote: post #6
justinph wrote: post #4

We're still using hypertext transfer protocol, right? You need to send hypertext down the wire. We shouldn't let one company, google, dictate how the web works, simply because of their proprietary technological innovation.

Roads are meant for horses, right? We shouldn't let one company, Ford, dictate how the roads work. My man, if we only used infrastructure and technology in the way it was originally intended and narrowly imagined, the world would be a dim place.

Ok, I see your point.

But, have Yahoo or Bing or DuckDuckGo made the transition to be able to crawl the web with a full JS & DOM rendering engine? I doubt it. By eschewing that compatibility we're setting a very high bar for what any competitor to google would have to achieve.

I like google. I just don't think it's good to have one company own a market so completely.

Re: Googlebot's recent improvements might revolutionize web development

#18

Post by mixonic »

andrenotgiant wrote: post #8

Taking a step back: The "Page" paradigm is still very much alive, despite these recent javascript parsing advances. 1. Google still needs a URL-addressable "PAGE" to which it can send Users. 2. This "PAGE" needs to be find-able via LINKS (javascript or HTML) and it needs to exist within a sensible hierarchy of a SITE. 3. This "PAGE" needs to have unique and significant content visible immediately to the user, and on…

I'd debate the phrase "step back". If you replace all your references to PAGE with URL, you get closer to a real meaning.

URLs for single-page applications are a serialization of application state. The fact that we now have an application platform (JavaScript/HTTP) providing sharable, mostly-human-readable state sharing (URLs) and is also indexed and searchable is nothing short of incredible.

Yes, the basic abstractions we use are the same. We will have URLs that address content in our applications. But now these are applications running on Google's own servers. Google is running my application (and hundreds of thousands more), and trying to understand what they mean to humans. This is a pretty amazing step forward.

Imagine Apple announcing it would run all iOS applications, interacting like a user to build a search index. IMO, this parallel shows what makes Google's commitment to running JavaScript apps exciting.

Re: Googlebot's recent improvements might revolutionize web development

#19

Post by iLoch »

jacquesm wrote: post #7

One potential problem here is that google will use this to widen the gap between it and the 'one page apps' web and other search engines (such as duckduckgo) that can't match it in resources. How strong of an advantage that will be in the long run is uncertain, I would rather see a web that ships pages with actual content in them than empty containers for a variety of reasons (most of which have to do with accessibil…

It's not difficult to set up middleware that'll render the page for any clients that require it. (For instance, we can assume any client that identifies as "bot" that's not Google probably wants a pre-rendered page, which we can do quite effortlessly. Here's one implementation for Nodejs: https://prerender.io, or you can always roll your own with something like Phantom.js.

Re: Googlebot's recent improvements might revolutionize web development

#20

Post by sigzero »

justinph wrote: post #17
o_____________o wrote: post #6

Earlier quoted context omitted.

Roads are meant for horses, right? We shouldn't let one company, Ford, dictate how the roads work. My man, if we only used infrastructure and technology in the way it was originally intended and narrowly imagined, the world would be a dim place.

Ok, I see your point. But, have Yahoo or Bing or DuckDuckGo made the transition to be able to crawl the web with a full JS & DOM rendering engine? I doubt it. By eschewing that compatibility we're setting a very high bar for what any competitor to google would have to achieve. I like google. I just don't think it's good to have one company own a market so completely.

I don't really like google and I agree.
Post reply on HN