Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: How Googlebot crawls JavaScript

#71

Post by debaserab2 »

userbinator wrote: post #39
compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. We're moving…

At the end of the day whether or not the source data is in a proprietary format or not, it's being rendered into HTML in the case of web apps. The only difference is whether that happens on the client or server side. It's trivial to parse in either case. In fact I'd argue it's usually easier to dig up a JSON end point in the case of a web app which is far more parseable than HTML.

I don't think we can point to this reason to explain the rise of web app's.

Re: How Googlebot crawls JavaScript

#72

Post by 15155 »

billybolero wrote: post #37
marrs wrote: post #33

Earlier quoted context omitted.

But you can still progressively enhance with JS to achieve that nice UI, and often it will be more usable because it's built on a solid RESTful foundation that is close to browser behaviour and therefore user expectation. My experience with JS only apps is that they're often less usable, more brittle, and often don't work at all in IE

Progressive enhancement work well for simple stuff. Like progressively enhancing a form post, or a "like" button which just sends an Ajax request. But as the complexity grows, progressive enhancement doesn't really scale and you end up with two separate versions of your site/app. I agree that Javascript only apps are often less usable, because the devs making them aren't testing enough on different browsers and devic…

> two separate versions of your site/app.

It isn't 2010 anymore.

React (just to name an example, there are many others) completely avoids this issue - you get serverside and clientside rendering out of the box.

Re: How Googlebot crawls JavaScript

#73

Post by [deleted] »

KMag wrote: post #54
kuschku wrote: post #23

Earlier quoted context omitted.

And how about server-side state? Imagine you use setInterval to load a new paragraph from a server, and the server only provides a new parapgraph 1200ms after the first polling?

Each time we re-analyzed the page, we got more data and more URLs for the crawler to grab and have waiting for us the next time we analyzed that particular page. Of course, session-dependent content would get badly messed up, but that generally doesn't make useful (or at least repeatable) search results anyway. Source: I primarily did JavaScript execution for Google's indexing pipeline 2006 to 2010.

[deleted]

Re: How Googlebot crawls JavaScript

#74

Post by blumkvist »

KMag wrote: post #43
netnichols wrote: post #26

Earlier quoted context omitted.

They probably don't care about that content. My first guess would be that they snapshot the DOM in the JS tick immediately after window.onload completes. Maybe they have a short pause to let any fast timeouts or callbacks complete, but there's got to be a cutoff at some point (e.g. to stop an infinite wait for pages that continuously update a relative date). Of course, with their own JS engine, I bet they can get rea…

Actually, we did care about this content. I'm not at liberty to explain the details, but we did execute setTimeouts up to some time limit. If they're smart, they actually make the exact timeout a function of a HMAC of the loaded source, to make it very difficult to experiment around, find the exact limits, and fool the indexing system. Back in 2010, it was still a fixed time limit. Source: executing JavaScript in Goo…

What about AJAX? Does it load/read/index data after the fact?

Re: How Googlebot crawls JavaScript

#75

Post by matthewmacleod »

userbinator wrote: post #39
compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. We're moving…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted.

I don't agree with this. Browsers are more user-targeted than ever.

We're moving from browsers being viewers for simple HTML documents (which can be copied, shared, and linked via simple means)

Browsers still allow this.

to a platform for running complex applications written in JavaScript which often render data retrieved in proprietary formats from proprietary APIs.

I have rarely seen a web API that uses anything other than straightforward JSON.

The "open by default" nature of plain HTML has become the "closed by default" of the data processed by web apps

Almost always equally as open as any HTML you would previously received.

Native app platforms (e.g. mobile) are also gradually becoming more "closed by default"; I'm not sure if that's a related trend.

What do you mean by this?

Re: How Googlebot crawls JavaScript

#76

Post by romaniv »

hrktb wrote: post #58
compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

I think thisviewpoint is too limited. 20 years ago a webpage was just text, but it has evolved in so much more. I'd be ok with a data site rendering everything from a set of json files. There is more legitimacy in having the presentation done in static html. Same would go with sites mixing different information sources (twitter, rss etc). You can do the data fetching server side, but the user might prefer having it d…

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

Re: How Googlebot crawls JavaScript

#77

Post by romaniv »

Just remember that there is life outside of Google and by brushing it off you're stifling further web innovations.

Also, I wonder how Google handles security while executing random JS code. It's one thing to hack into a single browser. It's another thing to hack into a crawler. Think of all the possibilities.

Re: How Googlebot crawls JavaScript

#78

Post by motti »

bhartzer wrote: post #63
amelius wrote: post #51

Earlier quoted context omitted.

Google made chrome, so I'm sure they know how to render webpages, and execute JavaScript. It's all mostly open source so you could use it too.

Yes, Google made Chrome. Keep in mind that they created Googlebot, and Chrome is really only a "slimmed down" version of Googlebot. Chrome came about because of Googlebot.

I don't think this is true at all.

Re: How Googlebot crawls JavaScript

#79

Post by MichaelGG »

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

Did you load in Ajax? I've got a client that runs a site that loads HTML in separately. They've been paying for a third party service to run PhantomJS and save HTML snapshots to serve to Googlebot - is that no longer needed?

(I'm not thrilled about rendering this way, but it makes development a lot easier.)

Re: How Googlebot crawls JavaScript

#80

Post by mcculley »

KMag wrote: post #52
mcculley wrote: post #35

I've long thought that the need for a high performance sandboxed JavaScript VM was the real impetus for Google's investment in v8, and that Chrome was just a useful opportunity to leverage it and to get external contribution. Is there any evidence that this is the case?

Unlikely. I was using SpiderMonkey to execute JavaScript in Google's indexing pipeline long before I had heard about v8, and I doubt Lars had me in mind when he started on v8. Of course, I tele-conferenced with Lars before Chrome was released, but SpiderMonkey was still the indexing system's JavaScript interpreter on Chrome's go-live date.

Interesting. Thanks for clarifying that!
Post reply on HN