Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

21–30 of 114 posts

Re: How Googlebot crawls JavaScript

#21

Post by andybak »

anaolykarpov wrote: post #5

I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()

A better example would be "how does Google index a page where an element changes sufficiently slowly that a user would see it post-change but a faster-than-real-time script would have a different experience"

Think of a long page where you need to read a few things as you scroll. You could game Google by timing it so that most humans would see content x but any script that ran at an unnatural speed would see y.

Re: How Googlebot crawls JavaScript

#23

Post by kuschku »

biokoda wrote: post #13
anaolykarpov wrote: post #5

I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()

Maybe they trigger timers immediately

And how about server-side state?

Imagine you use setInterval to load a new paragraph from a server, and the server only provides a new parapgraph 1200ms after the first polling?

Re: How Googlebot crawls JavaScript

#24

Post by fridek »

kuschku wrote: post #23
biokoda wrote: post #13

Earlier quoted context omitted.

Maybe they trigger timers immediately

And how about server-side state? Imagine you use setInterval to load a new paragraph from a server, and the server only provides a new parapgraph 1200ms after the first polling?

Waiting for all one-time async operations to finish is not unfeasible. Async loops (like pinging server for possible updates) is a tad harder.

Re: How Googlebot crawls JavaScript

#25

Post by bceagle »

There is no doubt Google continues to get better at indexing client side rendered HTML but it is not perfect and indexing is not the same as ranking high in organic search. For ranking, there are distinct advantages to server rendering. The biggest one is consistent initial page load performance. Long story short, if you care about ranking and not just indexing, you still need server rendering.

Re: How Googlebot crawls JavaScript

#26

Post by netnichols »

anaolykarpov wrote: post #5

I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()

They probably don't care about that content.

My first guess would be that they snapshot the DOM in the JS tick immediately after window.onload completes. Maybe they have a short pause to let any fast timeouts or callbacks complete, but there's got to be a cutoff at some point (e.g. to stop an infinite wait for pages that continuously update a relative date). Of course, with their own JS engine, I bet they can get really fancy with the heuristics to determine when to take that snapshot.

Re: How Googlebot crawls JavaScript

#27

Post by billybolero »

frik wrote: post #15

searchengineland hasn't tested AJAX as the author wrote in the comments: "That's a great question! Our test was to programmatically insert text where we wanted into the DOM, but not as a server side transaction, like AJAX."

I read another blog about someone who tested that (don't have the url, but it was easy to find), and their conclusion was that the crawler won't wait for any Ajax request to finish to let you render that content. If you want to render with Javascript, you need to make that data a part of the initial payload and render that data during onload.

Re: How Googlebot crawls JavaScript

#29

Post by onion2k »

erikb wrote: post #19
onion2k wrote: post #4

Earlier quoted context omitted.

I use nightwatch.js ( http://nightwatchjs.org/ ). It's a layer on top of Selenium that makes browser testing a lot more straightforward. If you start with small, straightforward tests and build testable things from there you code will improve.

The problem is that tester and developer aren't the same person. This way the tests are much better at finding expectations the developer didn't have, but it's harder to convince the developers to code more testable, because they don't know the pains of testing.

The point of testing is not to prove the code doesn't work. It's to prove the code does work. That subtle but important difference is the key to good testing.

Finding a problem with code is useful, but it's extremely limited. You might find 100 bugs, but if there's 101 bugs your product has the potential to fail completely. It's so much more useful to define a framework of things that the code has to do properly and make sure it does do them all properly. To that end, testing should come first - define what the code needs to do, write tests to make sure it does those things (automated unit tests where possible, but at the very least well defined processes for how you make sure it works), and then write the code to actually do it. Any developer who isn't interested in proving their code works, and will continue to work as it becomes more complex, is a terrible developer.

tl;dr If you want to fix testing don't write any code until you know how you're going to test it works.

Post reply on HN