How Googlebot crawls JavaScript • Hacker News Board

Re: How Googlebot crawls JavaScript

#11

Post by [deleted] » Tue, May 12, 2015, 10:02 AM UTC

compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

[deleted]

Re: How Googlebot crawls JavaScript

#12

Post by hpaavola » Tue, May 12, 2015, 10:07 AM UTC

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

    pip install robotframework-selenium2library

test.txt:

    *** Settings ***
    Library           Selenium2Library

    *** Test Cases ***
    My First Test
        [Setup]    Open Browser    http://google.com    firefox
        Input Text    name=q    Robot Framework
        Click Button    name=btnG
        Wait Until Page Contains Element    ires
        Click Link    Robot Framework
        [Teardown]    Close All Browsers

and then execute

    pybot test.txt

Selenium2 and Robot Framework are pretty neat when it comes to web testing.

Re: How Googlebot crawls JavaScript

#13

Post by biokoda » Tue, May 12, 2015, 10:12 AM UTC

anaolykarpov wrote: post #5

I wonder how google indexes a page that inserts an element in the DOM 120 seconds after the page was loaded using a setTimeout()

Maybe they trigger timers immediately

Re: How Googlebot crawls JavaScript

#14

Post by axx » Tue, May 12, 2015, 10:12 AM UTC

compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

IMHO: Websites that don't have "realtime" content should always stick with traditional HTML. I'm a Webdeveloper myself and i don't like the JavaScript Frontend trend.

Many Devs use Frontend JS in places where it's absolutely not needed. If you're building an App that updates in realtime, shows informations while it's created, i'm fine with Frontend JS, but it's an overkill for most content pages.

Sure, it depends on your implementation details, but as i said, it's just my opinion.

Re: How Googlebot crawls JavaScript

#15

Post by frik » Tue, May 12, 2015, 10:18 AM UTC

searchengineland hasn't tested AJAX as the author wrote in the comments: "That's a great question! Our test was to programmatically insert text where we wanted into the DOM, but not as a server side transaction, like AJAX."

Re: How Googlebot crawls JavaScript

#16

Post by LunaSea » Tue, May 12, 2015, 10:19 AM UTC

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

You should look into Phantom.js or Zombie.js.

Re: How Googlebot crawls JavaScript

#17

Post by maerten » Tue, May 12, 2015, 10:28 AM UTC

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

We're using CasperJS (basically a simple interface to PhantomJS), works great for us. Have a look here: http://casperjs.org/

Re: How Googlebot crawls JavaScript

#18

Post by riffraff » Tue, May 12, 2015, 10:31 AM UTC

compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

For those that ignore it, John Titor[0] was a time traveler sent back in time to acquire some obsolete IBM machine which is needed in the future to debug some legacy code.

[0] http://en.wikipedia.org/wiki/John_Titor

Re: How Googlebot crawls JavaScript

#19

Post by erikb » Tue, May 12, 2015, 10:34 AM UTC

onion2k wrote: post #4

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

I use nightwatch.js ( http://nightwatchjs.org/ ). It's a layer on top of Selenium that makes browser testing a lot more straightforward. If you start with small, straightforward tests and build testable things from there you code will improve.

The problem is that tester and developer aren't the same person. This way the tests are much better at finding expectations the developer didn't have, but it's harder to convince the developers to code more testable, because they don't know the pains of testing.

Re: How Googlebot crawls JavaScript

#20

Post by billybolero » Tue, May 12, 2015, 10:34 AM UTC

axx wrote: post #14

compbio wrote: post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

IMHO: Websites that don't have "realtime" content should always stick with traditional HTML. I'm a Webdeveloper myself and i don't like the JavaScript Frontend trend. Many Devs use Frontend JS in places where it's absolutely not needed. If you're building an App that updates in realtime, shows informations while it's created, i'm fine with Frontend JS, but it's an overkill for most content pages. Sure, it depends on…

I mostly agree, but at the same time, the rise of native apps has raised the bar of what people expect in terms of UX. Take Hacker News and Reddit, primarily content based sites and a good fit for the classic server rendered HTML approach. Still a lot of people prefer using native apps to access that content. You can only get so far by adding some CSS to make the site responsive, but you won't be anywhere near the UX that native gives you without JS.

If we want the nice UX without relying too heavily on Javascript, there's a lot that has to change on the HTML/CSS side of things. And I don't see that happening at all.

And let's not forget that you a) don't get accessibility for free just by rendering on the server and b) almost all screen readers today support Javascript.