Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

1–10 of 114 posts

Re: How Googlebot crawls JavaScript

#2

Post by onion2k »

With the way websites work today surely the only possible way to build a search engine is to make something like a headless browser (similar to PhantomJS) that crawls the web like a user, seeing what the user sees, ignoring everything that's hidden from the user, and interpreting the importance of pages like a user would. Just parsing the HTML source of the page won't even get close to seeing the key features of a page any more.

Impressive work by Google to do that at scale, of course, but they'd be dead in the water if they didn't.

Re: How Googlebot crawls JavaScript

#3

Post by erikb »

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

Re: How Googlebot crawls JavaScript

#4

Post by onion2k »

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

I use nightwatch.js ( http://nightwatchjs.org/ ). It's a layer on top of Selenium that makes browser testing a lot more straightforward. If you start with small, straightforward tests and build testable things from there you code will improve.

Re: How Googlebot crawls JavaScript

#7

Post by danielsamuels »

erikb wrote: post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

WebDriverWait and expected_conditions.presence_of_element_located are your friends. Here's some sample code from our automation project: https://gist.github.com/danielsamuels/a39e0fef4e15d2ab04b5

Re: How Googlebot crawls JavaScript

#9

Post by compbio »

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app.

A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons principles of progressive enhancement. Sometimes there is not even a fallback/graceful degradation, reminding me of these 2001-era: "Best viewed at 800x600 resolution in Netscape"-sites.

Google holds enormous clout among SEO's. Google says they will factor in site-speed and a large fraction of the web will become faster. Google can say more sternly that using JavaScript can have ugly consequences for user experience and accessibility, but they are swimming upstream: The web seems to be moving on to fancy new technologies regardless of what their SEO says.

Not much good comes from HTML5 JavaScript fans forcing your hand. Tor enabled JavaScript, because too much of the web would break without it, leading to a poor user experience. This led to a huge security gaffe, which I fully blame on webdevelopers eschewing basic principles, to get that slideshow running.

Post reply on HN