How Googlebot crawls JavaScript • Hacker News Board

Re: How Googlebot crawls JavaScript

#111

Post by miketuritzin » Thu, May 14, 2015, 4:05 AM UTC

KMag wrote: post #106

miketuritzin wrote: post #88

Earlier quoted context omitted.

As an aside, did you work on the indexing team at Google? I was on the indexing team from 2005-2007, and I remember that Javascript execution was being worked on then, but I don't remember who was doing it (was a long time ago ;) ). My name is my username.

I was always in the New York office (before and after the move from Times Square to Chelsea), on the Rich Content Team sub-team of Indexing. My username is the same as my old Google username. I was working on the lightweight high-performance JavaScript interpretation system that sandboxed pretty much just a JS engine and a DOM implementation that we could run on every web page on the index. Most of my work was trying…

Ah, okay, cool. Never visited the NY office. That's probably why I just remember the general idea that "JS execution was being worked on."

Re: How Googlebot crawls JavaScript

#112

Post by hrktb » Thu, May 14, 2015, 1:26 PM UTC

romaniv wrote: post #94

hrktb wrote: post #87

Earlier quoted context omitted.

I understand that everything in json could also be represented in another format. But if your master data is in json, does it always make sense to convert the data to static html just for the sake of it ? Would you build a server component only for that conversion ? The answer to these question will depend on your priorities and use case, and the choice can easily be between no site or a js rendered site.

You're assuming the same thing you're arguing for. Namely, that JSON should be your base format. I can equally say: "I understand that everything in HTML could also be represented in another format. But if your master data is in HTML, does it always make sense to convert the data to JSON just for the sake of it? Would you build a separate client-side rendering component only for that conversion?" There two non-circul…

I am arguing that no one format SHOULD be your base format for data. You'll happen to have a format or another for any reason, it could be json, xml, csv or anything else.

Contrary to that you are positing that everything should always be based on or converted to plain html before serving to a browser. That's the point where we disagree.

Re: How Googlebot crawls JavaScript

#113

Post by aboodman » Sat, May 16, 2015, 1:37 PM UTC

Drezzor wrote: post #98

I get rendering HTML from JS, handling timeouts, even infinite scroll. But how in the world are they handling onMouseOver, and other mouse events? My best guess so far is reversing the code from the document.location events.

I think you are misunderstanding how this works. Google isn't "handling" any events at all, your webpage is. Google is instead the source of those events - it is simulating the role of a user.

So the bot loads your webpage into a headless browser and sends it a series of events to simulate a user interacting with it, and waits for navigation requests.

There is probably a whitelist of simulation behaviors:

  * mouseover, then click each  node
  * mouseover every pixel
  * mouseover, then change every  node
  * mouseover, then click every 
  etc...

Caveat: though I worked at Google when this work was being done, I was on a different team and don't have any inside knowledge - just speculating on an approach that would make sense.

Re: How Googlebot crawls JavaScript

#114

Post by KMag » Wed, May 20, 2015, 12:03 PM UTC

olalonde wrote: post #90

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

> At some point, some SEO figured out that random() was always returning 0.5. I'm not sure if anyone figured out that JavaScript always saw the date as sometime in the Summer of 2006, but I presume that has changed. I hope they now set the random seed and the date using a keyed cryptographic hash of all of the loaded javascript and page text, so it's deterministic but very difficult to game. I don't get why the rende…

Byte-for byte de-duping of search results is perfect and fairly cheap. Fuzzy de-duping is more expensive and imperfect. Users get really annoyed when a single query gives them several results that seem like near copies of the same page.

Tons of pages have minor modifications by JavaScript, and only a very small percentage have modifications done by JavaScript that result in JavaScript analysis resulting in improved search results.

So, if JavaScript analysis isn't deterministic, it has a small negative effect on the search results of many pages that offsets the positive effect it has on a small number of pages.