Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deprecating our AJAX crawling scheme

googlewebmastercentral.blogspot.com

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Deprecating our AJAX crawling scheme

#61

Post by logn »

elorant wrote: post #47
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

You can use one of the many headless browsers available. Selenium, phantomjs, phantomjs+casper, webkit, chromium, awesomium, name your poison. All are quite competent in rendering modern web pages. You don’t need to reinvent the wheel.

Also, if you want a headless browser that uses solely a JRE, my project is https://github.com/machinepublishers/jbrowserdriver

Re: Deprecating our AJAX crawling scheme

#62

Post by [deleted] »

elorant wrote: post #47
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

You can use one of the many headless browsers available. Selenium, phantomjs, phantomjs+casper, webkit, chromium, awesomium, name your poison. All are quite competent in rendering modern web pages. You don’t need to reinvent the wheel.

[deleted]

Re: Deprecating our AJAX crawling scheme

#63

Post by AnbeSivam »

logn wrote: post #61
elorant wrote: post #47

Earlier quoted context omitted.

You can use one of the many headless browsers available. Selenium, phantomjs, phantomjs+casper, webkit, chromium, awesomium, name your poison. All are quite competent in rendering modern web pages. You don’t need to reinvent the wheel.

Also, if you want a headless browser that uses solely a JRE, my project is https://github.com/machinepublishers/jbrowserdriver

Any idea how good java's nashorn is for this.

Re: Deprecating our AJAX crawling scheme

#64

Post by copsarebastards »

chc wrote: post #21
greglindahl wrote: post #12

Earlier quoted context omitted.

You should still be worried. Just because googlebot expensively evaluates JS for some websites doesn't mean it will evaluate JS for your brand-new website. You might get crawled a lot less deeply than if you had good content in your static pages.

By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites. Do you have some reason to doubt that?

> By abandoning their AJAX crawling scheme as described in the OP, they are essentially saying that they will evaluate JS for all sites.

No, they are not. If you even think that's possible you're fundamentally misunderstanding how search engines work.

Re: Deprecating our AJAX crawling scheme

#65

Post by logn »

AnbeSivam wrote: post #63
logn wrote: post #61

Earlier quoted context omitted.

Also, if you want a headless browser that uses solely a JRE, my project is https://github.com/machinepublishers/jbrowserdriver

Any idea how good java's nashorn is for this.

Nashorn isn't actually used in Java's WebView (which my project leverages). Nashorn is used elsewhere in the JRE and replaces Rhino from prior releases, but WebView has used something else entirely: JavaScriptCore. Details: http://stackoverflow.com/questions/30104124/what-javascript-...

But essentially on performance, it's comparable to a desktop browser but still slower than I'd like. Java 9 should support HTTP 2 and async HTTP by default, which might help. And I've been looking into short-cutting some of the in-memory rendering but haven't had any breakthroughs yet.

As far as JavaScriptCore engine specifically, it's the default in WebKit so there should be good performance data out there on it.

Re: Deprecating our AJAX crawling scheme

#66

Post by tracker1 »

mixonic wrote: post #59
m0th87 wrote: post #30

Don't believe the hype. Google has been saying that they can execute javascript for years. Meanwhile, as far as I can see, most non-trivial applications still aren't being crawled successfully, including my company's. We recently got rid of prerender because of the promise from the last article from google saying the same thing [1]. It didn't work. 1: http://googlewebmastercentral.blogspot.com/2014/05/understan...

I noticed our companies Ember.js based SPA site not being indexed well until I added a sitemap. Then it quickly appeared in the rankings. Historically Google has been using some fork of Chrome 10 when indexing. I'm unsure what impact that is having on the reliability of app rendering, but I also trust the Google search team has done reasonable checks ensuring common sites and frameworks render correctly. I strongly s…

It's also worth noting that even when they do get JS delivered content, it updates much less frequently...

Then again, I really like react+redux+koa (r3k) for client-server rendering.... Hoping to do something more serious with it in the next few months at work.

Re: Deprecating our AJAX crawling scheme

#67

Post by AnbeSivam »

logn wrote: post #65
AnbeSivam wrote: post #63

Earlier quoted context omitted.

Any idea how good java's nashorn is for this.

Nashorn isn't actually used in Java's WebView (which my project leverages). Nashorn is used elsewhere in the JRE and replaces Rhino from prior releases, but WebView has used something else entirely: JavaScriptCore. Details: http://stackoverflow.com/questions/30104124/what-javascript-... But essentially on performance, it's comparable to a desktop browser but still slower than I'd like. Java 9 should support HTTP 2 an…

Thanks

Re: Deprecating our AJAX crawling scheme

#68

Post by tracker1 »

suneilp wrote: post #34
cotillion wrote: post #8

So they're actually evaluating all js and css Googlebot is consuming. That's insane. Can we forget about any new competitors in search engine land now? Not only do you have to match Google in relevance you'll actually have to implement your own BrowserBot just to download the pages.

PhantomJS allows you to render a page and fully manipulate or search it. It's a headless WebKit browser you can use from the command line and it works pretty well. Google is obviously doing the same thing. They even used to show images of what a url looks like in the search results. They stopped doing that as I suspect it uses up a lot of resources of many sites.

I can say that Bing definitely does do JS interpretation as part of some of their renderings... I switched the URL routing in a relatively large site (about 300k routes, including navigatable search urls), so that they were all consistent, and all pointing to the new routes via permanent redirect... previously the project was supporting all of their older routing schemes over time, and it was troublesome wrt SEO (duplicated content on many pages, or the same because search parameters were the same, but different structure, same for individual content pages). When the change happened, we saw a huge uptick in google analytics hits (one page, no clickthroughs) coming from two locations... both turned out to be MS data centers. It was a relatively common problem.

It was always just a little white noise in the past, but when suddenly a couple hundred thousand pages permanently redirect... it was interesting.

Re: Deprecating our AJAX crawling scheme

#69

Post by tracker1 »

anon1mous wrote: post #41
cmg wrote: post #23

Earlier quoted context omitted.

They're starting to consider page load speed as a factor in rankings, which would lead me to believe that they're letting all the social buttons / trackers / media load.

How do you know a page has loaded? A complex page with ads, AJAX, WebSockets may be constantly busy. Most social buttons, ads, etc. are now loaded by callbacks, that usually finish after the page has rendered.

Most of that data flow, barring user interaction is much more limited compared to the initial load of controls, iframes, images, etc... you can visibly see the drop off..

If you look at the network tab in chrome dev tools, you can see when the dom ready event fires, the window load event, and when it really feels the content was done loading. That final load time is when the data flow lulls out for a bit.

Re: Deprecating our AJAX crawling scheme

#70

Post by tracker1 »

Figs wrote: post #14
rdoherty wrote: post #3

Wow, I built a project that rendered JS built webpages for search engines via NodeJS and PhantomJS. Rendering webpages is extremely CPU intensive, I'm amazed at the amount of processing power Google must have to do this at Internet scale. I really hope this works, lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that.

> lots of JS libraries expect things like viewport and window size information, I wonder how Google is achieving that. Just plug in common screen parameters (e.g. 1920x1080, 1366x768, ...) and analyze it as if it were the result you'd get by default with Chrome on such a screen, I would imagine.

Same goes for user agent spoofing (to some extent). You can imagine most of the stuff when you use the chrome dev tools being done without actual user interaction.
Post reply on HN