Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

61–70 of 114 posts

Re: How Googlebot crawls JavaScript

#61
post #46
post #34

Earlier quoted context omitted.

You're probably not a native speaker of English, but of Latin. In Latin, `ignorare` can mean `not to know` in addition to its meaning of `not to pay attention to`, but in English, it only has the meaning of `not to pay attention to`. Vale.

There are most probably no native speakers of Latin. Maybe you meant Spanish?

How about time travellers from the Imperium Romanum?

Re: How Googlebot crawls JavaScript

#62
post #46
post #34

Earlier quoted context omitted.

You're probably not a native speaker of English, but of Latin. In Latin, `ignorare` can mean `not to know` in addition to its meaning of `not to pay attention to`, but in English, it only has the meaning of `not to pay attention to`. Vale.

There are most probably no native speakers of Latin. Maybe you meant Spanish?

I'm sure he meant speakers of Latin-derived languages. I see native French speakers make this mistake often in English.

Re: How Googlebot crawls JavaScript

#63
post #51
post #3

But how! I don't know about other people here, but in our company we haven't figured out how to parse (for testing of course) dynamic websites. All tools, including free tools like Selenium and paid tools like QF-Test, seem to not be able to understand how it works, or our web developers are not able to code dynamic web like it should be coded.

Google made chrome, so I'm sure they know how to render webpages, and execute JavaScript. It's all mostly open source so you could use it too.

Yes, Google made Chrome. Keep in mind that they created Googlebot, and Chrome is really only a "slimmed down" version of Googlebot. Chrome came about because of Googlebot.

Re: How Googlebot crawls JavaScript

#64
post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

> A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Not to sound completely apathetic, but so what? Most of us aren't building sites that we expect to be around in 10 years, much less 1000 years. The ephemeral nature of what we're building isn't lost on us - we're tradin…

The arguable reason the web exploded in the first place were the architectural principles behind it were intentionally constrained to enable 50+ year sustainability and recombination for apps built within its architecture.

This isn't so much about plain-jane HTML pages (useful as they are, since they have a simple interaction model than many understand and enjoy). It's more about using and exposing data in a visible manner (known formats and semantics) and hyperlinks rather than a single page app with opaque data. This gives you network effects.

Think about the minor uproar over hash-bang URLs around 5 years ago, Twitter being the primary offender. That was single page application oriented rather than hyperlink orientation. There is a reason they've moved away from that.

In the 90s, Google or Yahoo was just something students did with the links that were out there - that eventually generated hundreds of billions in value because of network effects and visbility of the information in HTML (Ie. They could apply algorithms to it like PageRank).

The point of the web architecture is that it enables serendipity. Most anyone who has had massive success in business will explain the role of luck, serendipity, and network effects in their rise.

Designing a web app for today's paycheck by closing it off behind a WebSocket+ JavaScript mess eliminates a proven avenue for network effects. Sometimes that might be OK, but it's unnecessarily limiting for many kinds of ventures.

Re: How Googlebot crawls JavaScript

#65
We wrote our scraper to use phantomjs via the selenium.webdriver interface in python 2, simply because for something like 80% of the sites we extract information from, the data was not fully available unless we could render the dynamic parts of the page. I am not at all surprised that Google's bot is executing js. I have assumed they could do this for years now.

As for pure html front-ends, I understand the attraction, but when a single js-based implementation gets you consistent behavior and presentation across all browsers and mobile devices the advantages are pretty huge.

Re: How Googlebot crawls JavaScript

#66
post #46

Earlier quoted context omitted.

There are most probably no native speakers of Latin. Maybe you meant Spanish?

I'm sure he meant speakers of Latin-derived languages. I see native French speakers make this mistake often in English.

Thanks for this information.

Re: How Googlebot crawls JavaScript

#69
post #14

Earlier quoted context omitted.

IMHO: Websites that don't have "realtime" content should always stick with traditional HTML. I'm a Webdeveloper myself and i don't like the JavaScript Frontend trend. Many Devs use Frontend JS in places where it's absolutely not needed. If you're building an App that updates in realtime, shows informations while it's created, i'm fine with Frontend JS, but it's an overkill for most content pages. Sure, it depends on…

I mostly agree, but at the same time, the rise of native apps has raised the bar of what people expect in terms of UX. Take Hacker News and Reddit, primarily content based sites and a good fit for the classic server rendered HTML approach. Still a lot of people prefer using native apps to access that content. You can only get so far by adding some CSS to make the site responsive, but you won't be anywhere near the UX…

hacker news and reddit are very close to real time content.

reddit is almost a chat service.

your example is also very poignant as neither of those would get close to exist without proper urls

Re: How Googlebot crawls JavaScript

#70
post #9

I feel that dynamic websites are not websites, but applications. Even after this thorough research, I'd still be very wary of turning a primarily content-based site into a dynamic app. A plain HTML site is accessible, and will be accessible in a 1000 years. A site depending on (external) JavaScript sources will force John Titor to travel back in time to find version 1.x of jQuery. Starting with JavaScript abandons pr…

While I cannot speak for anyone else, I rarely ever use JavaScript and my "user experience" not at all "degraded". The browser I use, along with netcat, tcpclient, etc., does not even support JavaScript.

The only exceptions are when sites force use of JavaScript. Curiously, often these are sites where money is involved, e.g., banks, merchants, etc. I guess JavaScript makes things safer in those cases? Then one of the popular high complexity (and high security of course) browsers becomes necessary.

Perhaps it is because the type of content I consume is just reading material, listening material or viewing material. Each of which I can usually download and view with a dedicated application, if I so choose.

I used to think JavaScript might become unavoidable for the user and I spent time thinking about how to accomodate it. I often pondered how search engines would cope with it as well. But over the years I have changed my mind; I do not spend any time worrying about JavaScript as a barrier to content.

JavaScript can be a nuisance for non-interactive www usage but, at least in my experience, with some effort this impediment can be overcome. Whether the "user" is a nerd using nc or Googlebot.

Do you think 1000 years from now there will still be one group of people working to make the www more "interactive" and another group of people working to make the www more "machine readable", the later undoing the work of the former?

Post reply on HN