Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: How Googlebot crawls JavaScript

#81

Post by magicalist »

15155 wrote: post #72
billybolero wrote: post #37

Earlier quoted context omitted.

Progressive enhancement work well for simple stuff. Like progressively enhancing a form post, or a "like" button which just sends an Ajax request. But as the complexity grows, progressive enhancement doesn't really scale and you end up with two separate versions of your site/app. I agree that Javascript only apps are often less usable, because the devs making them aren't testing enough on different browsers and devic…

> two separate versions of your site/app. It isn't 2010 anymore. React (just to name an example, there are many others) completely avoids this issue - you get serverside and clientside rendering out of the box.

You dropped the context of that quote. React isn't exactly a poster child of progressive enhancement.

Re: How Googlebot crawls JavaScript

#82

Post by kyle6884 »

Loic wrote: post #32

For people wondering about Ajax requests, Googlebot is performing them very well together with SVG rendering. For example this URL: https://www.chemeo.com/predict?smiles=CCCC is performing the drawing of the molecule using RaphaelJS, then pulling the corresponding molecule from the database using Ajax and updating the page. Googlebot is performing all the steps perfectly well to add the end index the page. It is very…

you could add an additional URL parameter via pushState and ensure that you're defining the canonical tag only to the main data pages. You could also define the new parameter in webmaster tools and tell googlebot to ignore it

Re: How Googlebot crawls JavaScript

#83

Post by copsarebastards »

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

> (This is when I learned about holding down shift while hitting the browser's reload button to make it act like it was a fresh page fetch.)

Most useful aside of all time.

Re: How Googlebot crawls JavaScript

#84

Post by bovermyer »

userbinator wrote: post #39
compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. We're moving…

Surely you're not suggesting that parsing HTML is a good way to retrieve data for display?

Re: How Googlebot crawls JavaScript

#85

Post by tomjen3 »

rimantas wrote: post #53
axx wrote: post #14

Earlier quoted context omitted.

IMHO: Websites that don't have "realtime" content should always stick with traditional HTML. I'm a Webdeveloper myself and i don't like the JavaScript Frontend trend. Many Devs use Frontend JS in places where it's absolutely not needed. If you're building an App that updates in realtime, shows informations while it's created, i'm fine with Frontend JS, but it's an overkill for most content pages. Sure, it depends on…

I like Tantek's definition the best: "if it’s not curlable, it’s not on the web". http://tantek.com/2015/069/t1/js-dr-javascript-required-dead

That just sounds like a passive-aggressive arbitrary rule change. For about 99% of the world the curability, or not, of the web doesn't matter at all.

We already have a perfectly good web and includes things that are not curable, even if we exclude javascript (trivial example: you can't, meaningfully, curl a live sport event).

Re: How Googlebot crawls JavaScript

#86

Post by tomjen3 »

romaniv wrote: post #76
hrktb wrote: post #58

Earlier quoted context omitted.

I think thisviewpoint is too limited. 20 years ago a webpage was just text, but it has evolved in so much more. I'd be ok with a data site rendering everything from a set of json files. There is more legitimacy in having the presentation done in static html. Same would go with sites mixing different information sources (twitter, rss etc). You can do the data fetching server side, but the user might prefer having it d…

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

CSS doesn't even have a way to select all h1 elements that contains a div with a date class so I strongly question the assertion that it has "extremely powerful CSS queries" (if you think you are about to prove me wrong with a one-liner, please re-read the phrasing).

Re: How Googlebot crawls JavaScript

#87

Post by hrktb »

romaniv wrote: post #76
hrktb wrote: post #58

Earlier quoted context omitted.

I think thisviewpoint is too limited. 20 years ago a webpage was just text, but it has evolved in so much more. I'd be ok with a data site rendering everything from a set of json files. There is more legitimacy in having the presentation done in static html. Same would go with sites mixing different information sources (twitter, rss etc). You can do the data fetching server side, but the user might prefer having it d…

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

I understand that everything in json could also be represented in another format. But if your master data is in json, does it always make sense to convert the data to static html just for the sake of it ? Would you build a server component only for that conversion ?

The answer to these question will depend on your priorities and use case, and the choice can easily be between no site or a js rendered site.

Re: How Googlebot crawls JavaScript

#88

Post by miketuritzin »

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

As an aside, did you work on the indexing team at Google? I was on the indexing team from 2005-2007, and I remember that Javascript execution was being worked on then, but I don't remember who was doing it (was a long time ago ;) ). My name is my username.

Re: How Googlebot crawls JavaScript

#89

Post by snowwrestler »

copsarebastards wrote: post #83
KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

> (This is when I learned about holding down shift while hitting the browser's reload button to make it act like it was a fresh page fetch.) Most useful aside of all time.

I used to use this a lot. My experience is that for some reason, a couple years ago it stopped working reliably as a fresh page fetch. Some items were still coming up cached. Now I use incognito or private browsing windows instead.

Re: How Googlebot crawls JavaScript

#90

Post by olalonde »

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

> At some point, some SEO figured out that random() was always returning 0.5. I'm not sure if anyone figured out that JavaScript always saw the date as sometime in the Summer of 2006, but I presume that has changed. I hope they now set the random seed and the date using a keyed cryptographic hash of all of the loaded javascript and page text, so it's deterministic but very difficult to game.

I don't get why the rendering had to be deterministic. Server-side rendered HTML documents can also contain random data and it doesn't seem to prevent Google from doing "duplicate elimination".

Post reply on HN