Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

41–50 of 114 posts

Re: How Googlebot crawls JavaScript

#41

Post by KMag »

INTPenis wrote: post #28

I'm just waiting for the first security researcher to exploit the googlebot.

My primary role at Google from 2006 to 2010 was executing JavaScript in the indexing pipeline (not exactly in Googlebot, but close enough). I knew I was executing probably every known exploit out there, plus a lot of 0-days, and took lots of precautions (single-threaded subprocesss with very restricted list of allowed syscalls, etc.). It's not perfect, but breaking out of the sandbox would require a kernel 0-day in the subsystem used by our sandbox, plus a JS engine exploit.

Re: How Googlebot crawls JavaScript

#42

Post by mikesaloxi »

hughw wrote: post #31

Now, what kinds of V8 vulnerabilities can we exploit to get inside Google? Said every intelligence agency everywhere.

> Said every intelligence agency everywhere.

Not everywhere. The ones in the USA already have free access to anything in Google. In fact, Google is a part of their network.

Re: How Googlebot crawls JavaScript

#43

Post by KMag »

netnichols wrote: post #26
anaolykarpov wrote: post #5

I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()

They probably don't care about that content. My first guess would be that they snapshot the DOM in the JS tick immediately after window.onload completes. Maybe they have a short pause to let any fast timeouts or callbacks complete, but there's got to be a cutoff at some point (e.g. to stop an infinite wait for pages that continuously update a relative date). Of course, with their own JS engine, I bet they can get rea…

Actually, we did care about this content. I'm not at liberty to explain the details, but we did execute setTimeouts up to some time limit.

If they're smart, they actually make the exact timeout a function of a HMAC of the loaded source, to make it very difficult to experiment around, find the exact limits, and fool the indexing system. Back in 2010, it was still a fixed time limit.

Source: executing JavaScript in Google's indexing pipeline was my job from 2006 to 2010.

Re: How Googlebot crawls JavaScript

#45

Post by Loic »

gildas wrote: post #36
Loic wrote: post #32

For people wondering about Ajax requests, Googlebot is performing them very well together with SVG rendering. For example this URL: https://www.chemeo.com/predict?smiles=CCCC is performing the drawing of the molecule using RaphaelJS, then pulling the corresponding molecule from the database using Ajax and updating the page. Googlebot is performing all the steps perfectly well to add the end index the page. It is very…

However, if you search on Google: "Property Prediction for Butane" site:https://www.chemeo.com You'll see this page is not indexed.

Interesting, it looks like they massively dropped these pages from the index (or at least, from what they return as results) this is great (if they are not dropping other important pages)!

Re: How Googlebot crawls JavaScript

#46

Post by madez »

mdemare wrote: post #34
riffraff wrote: post #18

Earlier quoted context omitted.

For those that ignore it, John Titor[0] was a time traveler sent back in time to acquire some obsolete IBM machine which is needed in the future to debug some legacy code. [0] http://en.wikipedia.org/wiki/John_Titor

You're probably not a native speaker of English, but of Latin. In Latin, `ignorare` can mean `not to know` in addition to its meaning of `not to pay attention to`, but in English, it only has the meaning of `not to pay attention to`. Vale.

There are most probably no native speakers of Latin. Maybe you meant Spanish?

Re: How Googlebot crawls JavaScript

#47

Post by riffraff »

mdemare wrote: post #34
riffraff wrote: post #18

Earlier quoted context omitted.

For those that ignore it, John Titor[0] was a time traveler sent back in time to acquire some obsolete IBM machine which is needed in the future to debug some legacy code. [0] http://en.wikipedia.org/wiki/John_Titor

You're probably not a native speaker of English, but of Latin. In Latin, `ignorare` can mean `not to know` in addition to its meaning of `not to pay attention to`, but in English, it only has the meaning of `not to pay attention to`. Vale.

fair enough, assuming you mean "latin derived language" thanks!

Re: How Googlebot crawls JavaScript

#48

Post by arielb1 »

billybolero wrote: post #37
marrs wrote: post #33

Earlier quoted context omitted.

But you can still progressively enhance with JS to achieve that nice UI, and often it will be more usable because it's built on a solid RESTful foundation that is close to browser behaviour and therefore user expectation. My experience with JS only apps is that they're often less usable, more brittle, and often don't work at all in IE

Progressive enhancement work well for simple stuff. Like progressively enhancing a form post, or a "like" button which just sends an Ajax request. But as the complexity grows, progressive enhancement doesn't really scale and you end up with two separate versions of your site/app. I agree that Javascript only apps are often less usable, because the devs making them aren't testing enough on different browsers and devic…

I still prefer just-HTML sites to the typical JS-based sites (e.g. the new Google Groups) I see.

Re: How Googlebot crawls JavaScript

#49

Post by KMag »

This was actually my primary role at Google from 2006 to 2010.

One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language pages, my best guess is that they weren't trying to hide content from search engines, but rather trying to work around some old browser bug that incorrectly rendered (or made ugly) Chinese text, but somehow rendering text via JavaScript avoided the bug.

The really interesting parts were (1) trying to make sure that rendering was deterministic (so that identical pages always looked identical to Google for duplicate elimination purposes) (2) detecting when we deviated significantly from real browser behavior (so we didn't generate too many nonsense URLs for the crawler or too many bogus redirects), and (3) making the emulated browser look a bit like IE and Firefox (and later Chrome) at the some time, so we didn't get tons of pages that said "come back using IE" er "please download Firefox".

I ended up modifying SpiderMonkey's bytecode dispatch to help detect when the simulated browser had gone off into the weeds and was likely generating nonsense.

I went through a lot of trouble figuring out the order that different JavaScript events were fired off in IE, FireFox, and Chrome. It turns out that some pages actually fire off events in different orders between a freshly loaded page and a page if you hit the refresh button. (This is when I learned about holding down shift while hitting the browser's reload button to make it act like it was a fresh page fetch.)

At some point, some SEO figured out that random() was always returning 0.5. I'm not sure if anyone figured out that JavaScript always saw the date as sometime in the Summer of 2006, but I presume that has changed. I hope they now set the random seed and the date using a keyed cryptographic hash of all of the loaded javascript and page text, so it's deterministic but very difficult to game. (You can make the date determistic for a month and dates of different pages jump forward at different times by adding an HMAC of page content (mod number of seconds in a month) to the current time, rounding down that time to a month boundary, and then subtracting back the value you added earlier. This prevents excessive index churn from switching all dates at once, and yet gives each page a unique date.)

Re: How Googlebot crawls JavaScript

#50

Post by axx »

billybolero wrote: post #20
axx wrote: post #14

Earlier quoted context omitted.

IMHO: Websites that don't have "realtime" content should always stick with traditional HTML. I'm a Webdeveloper myself and i don't like the JavaScript Frontend trend. Many Devs use Frontend JS in places where it's absolutely not needed. If you're building an App that updates in realtime, shows informations while it's created, i'm fine with Frontend JS, but it's an overkill for most content pages. Sure, it depends on…

I mostly agree, but at the same time, the rise of native apps has raised the bar of what people expect in terms of UX. Take Hacker News and Reddit, primarily content based sites and a good fit for the classic server rendered HTML approach. Still a lot of people prefer using native apps to access that content. You can only get so far by adding some CSS to make the site responsive, but you won't be anywhere near the UX…

I agree! But you have to make a differnce between "enhancing the UX with JS" and "the whole App is written in JS".

I think it's perfectly fine to enhance stuff with JS, as long as important content is visible without it.

Some people go head over heels the JS route (since this seems the way people do it today) and build things that can be build way cheaper (measured in hours) with traditional HTML. Since the outcome is the same (static content) it's just not necessary.

Note: I'm focusing on static content here. Pages that mainly show text and images (Blogs, Newssites et al). (Web)Apps are another topic that present good reasons to use Frontend JS.

Post reply on HN