Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Googlebot crawls JavaScript

searchengineland.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: How Googlebot crawls JavaScript

#91

Post by laumars »

snowwrestler wrote: post #89
copsarebastards wrote: post #83

Earlier quoted context omitted.

> (This is when I learned about holding down shift while hitting the browser's reload button to make it act like it was a fresh page fetch.) Most useful aside of all time.

I used to use this a lot. My experience is that for some reason, a couple years ago it stopped working reliably as a fresh page fetch. Some items were still coming up cached. Now I use incognito or private browsing windows instead.

If you're running Chrom(e|ium) with developer tools open then you can right click the refresh button and it gives you a few refresh options (eg clear cache and reload).

That tends to be my fall back whenever I'm specifically fussed about the "freshness" of a page. That or curl

Re: How Googlebot crawls JavaScript

#92

Post by gwern »

KMag wrote: post #40
hughw wrote: post #31

Now, what kinds of V8 vulnerabilities can we exploit to get inside Google? Said every intelligence agency everywhere.

From 2006 to 2010, my primary role at Google was JavaScript execution in the indexing pipeline. I knew I was likely executing every known JavaScript engine exploit out there plus a good number of 0-days, and ran the javascript engine in a single-threaded subprocess with a greatly restricted set of allowed system calls. Certainly the right combination of kernel zero-days and JS interpreter exploits could be used to ta…

> ran the javascript engine in a single-threaded subprocess with a greatly restricted set of allowed system calls.

You were trying to sandbox the JS engine rather than using disposable VMs?

Re: How Googlebot crawls JavaScript

#93

Post by tracker1 »

MichaelGG wrote: post #79
KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

Did you load in Ajax? I've got a client that runs a site that loads HTML in separately. They've been paying for a third party service to run PhantomJS and save HTML snapshots to serve to Googlebot - is that no longer needed? (I'm not thrilled about rendering this way, but it makes development a lot easier.)

In practice, and from experience... content changes driven by JS tend to lag a few days, if the content was changed via direct output... If you're doing client-side rendering, couldn't you refactor to use node, or similar for your output rendering?

If you aren't heavy reliant on conversions from search traffic, you can probably get away with being JS driven, I'd suggest sticking with Anchor tags for direct navigation with JS overrides. Assuming you are supporting full url changes.. otherwise you need to support he shebang alternate paths... which is/was a pain when I did it 3-4 years ago.

Re: How Googlebot crawls JavaScript

#94

Post by romaniv »

hrktb wrote: post #87
romaniv wrote: post #76

Earlier quoted context omitted.

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

I understand that everything in json could also be represented in another format. But if your master data is in json, does it always make sense to convert the data to static html just for the sake of it ? Would you build a server component only for that conversion ? The answer to these question will depend on your priorities and use case, and the choice can easily be between no site or a js rendered site.

You're assuming the same thing you're arguing for. Namely, that JSON should be your base format. I can equally say:

"I understand that everything in HTML could also be represented in another format. But if your master data is in HTML, does it always make sense to convert the data to JSON just for the sake of it? Would you build a separate client-side rendering component only for that conversion?"

There two non-circular considerations that favor HTML:

1. Web pages still render via DOM. JSON data has to be tramsformed, HTML does not.

2. HTML has semantic capabilities. JSON does not.

Re: How Googlebot crawls JavaScript

#95

Post by tracker1 »

axx wrote: post #50
billybolero wrote: post #20

Earlier quoted context omitted.

I mostly agree, but at the same time, the rise of native apps has raised the bar of what people expect in terms of UX. Take Hacker News and Reddit, primarily content based sites and a good fit for the classic server rendered HTML approach. Still a lot of people prefer using native apps to access that content. You can only get so far by adding some CSS to make the site responsive, but you won't be anywhere near the UX…

I agree! But you have to make a differnce between "enhancing the UX with JS" and "the whole App is written in JS". I think it's perfectly fine to enhance stuff with JS, as long as important content is visible without it. Some people go head over heels the JS route (since this seems the way people do it today) and build things that can be build way cheaper (measured in hours) with traditional HTML. Since the outcome i…

It depends... if your content is mostly static, or mostly dynamic it really depends...

Sometimes doing it all in react/angular is easier than bolting on jquery extension after extension... bloating everything up. Also, if you're using more modern techniques, you're going through a build/minify step anyways which makes it even easier still to be more JS based than static.

Re: How Googlebot crawls JavaScript

#96

Post by romaniv »

tomjen3 wrote: post #86
romaniv wrote: post #76

Earlier quoted context omitted.

90% of everything served as JSON can be served as semantic HTML and then manipulated with roughly the same amount of code required to manipulate JSON. Yes, JSON navigation is "built in". However, HTML has incredibly powerful CSS queries which allow you to manipulate hierarchical data with minimal fuss.

CSS doesn't even have a way to select all h1 elements that contains a div with a date class so I strongly question the assertion that it has "extremely powerful CSS queries" (if you think you are about to prove me wrong with a one-liner, please re-read the phrasing).

You did not provide an example of production-ready JSON query library for feature comparison.

Re: How Googlebot crawls JavaScript

#97

Post by snowwrestler »

laumars wrote: post #91
snowwrestler wrote: post #89

Earlier quoted context omitted.

I used to use this a lot. My experience is that for some reason, a couple years ago it stopped working reliably as a fresh page fetch. Some items were still coming up cached. Now I use incognito or private browsing windows instead.

If you're running Chrom(e|ium) with developer tools open then you can right click the refresh button and it gives you a few refresh options (eg clear cache and reload). That tends to be my fall back whenever I'm specifically fussed about the "freshness" of a page. That or curl

Thanks, never tried right-clicking that before. There's also a checkbox in dev tools settings to "Disable cache while dev tools is open."

Re: How Googlebot crawls JavaScript

#98

Post by Drezzor »

I get rendering HTML from JS, handling timeouts, even infinite scroll. But how in the world are they handling onMouseOver, and other mouse events? My best guess so far is reversing the code from the document.location events.

Re: How Googlebot crawls JavaScript

#99

Post by findjashua »

copsarebastards wrote: post #83
KMag wrote: post #49

This was actually my primary role at Google from 2006 to 2010. One of my first test cases was a certain date range of the Wall Street Journal's archives of their Chinese language pages, where all of the actual text was in a JavaScript string literal, and before my changes, Google thought all of these pages had identical content... just the navigation boilerplate. Since the WSJ didn't do this for its English language…

> (This is when I learned about holding down shift while hitting the browser's reload button to make it act like it was a fresh page fetch.) Most useful aside of all time.

i used to go through a lot of head scratching when doing manual testing, before discovering the joys of cmd-shift-r

Re: How Googlebot crawls JavaScript

#100

Post by scintill76 »

matthewmacleod wrote: post #75
userbinator wrote: post #39

Earlier quoted context omitted.

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. We're moving…

I think the trend of "turning a primarily content-based site into a dynamic app", and indeed most of what has been referred to as "Web progress", "moving the Web forward", etc. comes from the desire of content producers to obtain and maintain more control over their content. Look at how browsers have evolved to de-emphasise features which give the user control while adding those that are author-targeted. I don't agre…

> I have rarely seen a web API that uses anything other than straightforward JSON.

There are differences between HTML vs. AJAX+JSON+Javascript+DOM. JSON has a lot less of a schema than HTML. You don't have to execute custom code from a remote server to render plain HTML. Client-side rendering is more complex for the client. A Javascript-based page is typically going to require more requests for remote resources than inline HTML, potentially meaning more bandwidth and caching/archiving costs. I can't quite put my finger on the implications right now, but I wanted to note that "JSON is a standard." doesn't mean much to me, since JSON is not comparable to HTML.

Post reply on HN