Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: How Googlebot crawls JavaScript

#101

Post by mark_l_watson »

hrktb wrote: post #87
romaniv wrote: post #76

Earlier quoted context omitted.

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

I understand that everything in json could also be represented in another format. But if your master data is in json, does it always make sense to convert the data to static html just for the sake of it ? Would you build a server component only for that conversion ? The answer to these question will depend on your priorities and use case, and the choice can easily be between no site or a js rendered site.

I interpret 'master data' as meaning something different: a uniform set of schemes used throughout an enterprise that supports an organization's data stores. Master data is a canonical representation.

Re: How Googlebot crawls JavaScript

#102

Post by e12e »

matthewmacleod wrote: post #75
userbinator wrote: post #39

Earlier quoted context omitted.

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. We're moving…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. I don't agre…

>> We're moving from browsers being viewers for simple HTML documents (which can be copied, shared, and linked via simple means)

> Browsers still allow this.

Yes, they allow it. But consider that user stylesheets have been dropped, hardly any improvements have been made with presenting standard html (unless you count giving in to IE and moving from less-stark black-on-grey to too stark black-on-white as an improvement).

Still no browser does a half-decent job of avoiding ragged-right text[1], give you decent margins on un-styled content, etc. There's no real reason for this. You could claim "backwards compatability" -- but if there was genuine interest, there'd be nothing stopping the introduction of a content type.

It's ironic, that browsing a plain html-site in w3m in the console is a better reading experience than opening the same page in a desktop browser. So of course people need to supply a crap-load of stuff just to get decent, basic text layout, that flows well across various screen sizes. No reason a basic, unstyled html-document couldn't look much better than a TeX/LaTeX document published in the 80s, with the added bonus of re-flowing in a sane way for various window/screen sizes -- but they all look awful, too the point that plain html is actually not usable.

You need to wrap a document in js to get sensible layout, and in css to get sensible presentation. Even if the document doesn't contain any other media than text. Add an image or two, and things keep going downhill. It's absurd.

[1] http://alistapart.com/article/the-look-that-says-book

Re: How Googlebot crawls JavaScript

#103

Post by e12e »

tomjen3 wrote: post #85
rimantas wrote: post #53

Earlier quoted context omitted.

I like Tantek's definition the best: "if it’s not curlable, it’s not on the web". http://tantek.com/2015/069/t1/js-dr-javascript-required-dead

That just sounds like a passive-aggressive arbitrary rule change. For about 99% of the world the curability, or not, of the web doesn't matter at all. We already have a perfectly good web and includes things that are not curable, even if we exclude javascript (trivial example: you can't, meaningfully, curl a live sport event).

What do you mean, can't curl a live sport event? Do you mean you can't save it to disk (PVR), or if it's not audio/video, it's not useful to curl it and parse it for stats? If it is video, and you don't consider curl-as-pvr a valid use-case of curl'ing -- how about presenting the text-overlay as a text/html/rss-feed? Mix it at the client for those that want video, or show it as an RSS stream for text w/images?

Not to mention that when we demand to have a functional js-parser and dom-tree just to get at the content, things like text-to-speech and many other things (including building a search-engine!) becomes much harder. For very little (I'd say no) gain.

Re: How Googlebot crawls JavaScript

#104

Post by erikb »

bhartzer wrote: post #63
amelius wrote: post #51

Earlier quoted context omitted.

Google made chrome, so I'm sure they know how to render webpages, and execute JavaScript. It's all mostly open source so you could use it too.

Yes, Google made Chrome. Keep in mind that they created Googlebot, and Chrome is really only a "slimmed down" version of Googlebot. Chrome came about because of Googlebot.

Or maybe they did it to get full access to the users (having access to your customers through other providers sks), get a more stable web experience (which increases the numbers of all users and thereby the number of google ad viewers), and have more power in the definition of web standards?

Re: How Googlebot crawls JavaScript

#106

Post by KMag »

miketuritzin wrote: post #88
KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

As an aside, did you work on the indexing team at Google? I was on the indexing team from 2005-2007, and I remember that Javascript execution was being worked on then, but I don't remember who was doing it (was a long time ago ;) ). My name is my username.

I was always in the New York office (before and after the move from Times Square to Chelsea), on the Rich Content Team sub-team of Indexing. My username is the same as my old Google username.

I was working on the lightweight high-performance JavaScript interpretation system that sandboxed pretty much just a JS engine and a DOM implementation that we could run on every web page on the index. Most of my work was trying to improve the fidelity of the system. My code analyzed every web page in the index.

Towards the end of my time there, there was someone in Mountain View working on a heavier, higher-fidelity system that sandboxed much more of a browser, and they were trying to improve performance so they could use it on a higher percentage of the index.

Re: How Googlebot crawls JavaScript

#107

Post by ipullrank »

KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

Great thread. I'm one of those people that was poking around and trying to figure this out a few years back.

Obviously, there's a lot I know you can't say, but I'd love to know your general thoughts on how far off we were: http://ipullrank.com/googlebot-is-chrome https://moz.com/blog/just-how-smart-are-search-robots

Re: How Googlebot crawls JavaScript

#108

Post by mdemare »

riffraff wrote: post #47
mdemare wrote: post #34

Earlier quoted context omitted.

You're probably not a native speaker of English, but of Latin. In Latin, `ignorare` can mean `not to know` in addition to its meaning of `not to pay attention to`, but in English, it only has the meaning of `not to pay attention to`. Vale.

fair enough, assuming you mean "latin derived language" thanks!

Actually I assumed you were a time traveller.

Re: How Googlebot crawls JavaScript

#109

Post by marrs »

magicalist wrote: post #81
15155 wrote: post #72

Earlier quoted context omitted.

> two separate versions of your site/app. It isn't 2010 anymore. React (just to name an example, there are many others) completely avoids this issue - you get serverside and clientside rendering out of the box.

You dropped the context of that quote. React isn't exactly a poster child of progressive enhancement.

Actually it's very well suited to it (well, in theory at least - haven't tried it yet). You can render the same html on the server as on the client, using node.js, so you can build a page on the server and then let the JS enabled client take over after the initial page load, or let the server do the work for the non-JS enabled browser.

Re: How Googlebot crawls JavaScript

#110

Post by marrs »

billybolero wrote: post #37
marrs wrote: post #33

Earlier quoted context omitted.

But you can still progressively enhance with JS to achieve that nice UI, and often it will be more usable because it's built on a solid RESTful foundation that is close to browser behaviour and therefore user expectation. My experience with JS only apps is that they're often less usable, more brittle, and often don't work at all in IE

Progressive enhancement work well for simple stuff. Like progressively enhancing a form post, or a "like" button which just sends an Ajax request. But as the complexity grows, progressive enhancement doesn't really scale and you end up with two separate versions of your site/app. I agree that Javascript only apps are often less usable, because the devs making them aren't testing enough on different browsers and devic…

It often is. Most web apps are business applications, doing CRUD - form filling, and if you were around before AJAX, you'll know that we were building these back then just fine. I'm not saying we can't do better now, but that PE will get you to a better experience than graceful degredation.

Sometimes you need additional functionality, maybe a realtime graph of share prices that has to be JS. So progressively enhance just that component, or gracefully degrade if you have to (e.g. put a sign up saying "switch on JS to get this specific functionality") but don't use it as an excuse to turn everyone away. It might be that you will reach people who can live without the stock ticker.

Sometimes you just can't do without JS. I wrote a desktop publishing app on the browser once. Obviously I wrote it in JS - users were forced to use a modern standards compliant browser (this was an internal app) - but if I'm doing an ecommerce site, or really any public site, I'm always challenging the devs who want to "build it in angular" to reconsider that option before ploughing ahead.

Post reply on HN