Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Javascript apps can be fully crawlable

#101

Post by uptown »

dchest wrote: post #61
commanderj wrote: post #56

Making JS heavy sites crawlable is also possible with libraries like https://github.com/minddust/jquery-pjaxr and https://github.com/defunkt/jquery-pjax . Plus the push state has the advantage of "real" urls.

How?

With each user-interaction that updates a page fragment it modifies the address in the browser's address bar to correspond to the current state. If somebody were to copy and paste that URL into a new tab, your site would load the complete interface if you've structured your back-end code correctly.

You do this by building in logic to the part of the code that outputs your view to see whether the request is coming as a PJAX request, or not. If it is, you output the page-fragment, which is then added to your existing DOM. If it's not a PJAX request, your back-end outputs the entire code for your site.

There's a limitation to PJAX where you can only update one fragment at a time, though PJAXR seems to address that limitation by providing support for updating multiple-fragments simultaneously. Either way, you get the huge advantage of having a fully-crawlable site without needing to integrate pre-rendering work-arounds for search-engine compatibility.

Re: Javascript apps can be fully crawlable

#105

Post by se_ »

If you're using Rails have a look at https://github.com/seojs/seojs-ruby, it's a gem similar to prerender but it's using our managed service at http://getseojs.com/ to get the snapshots. There are also ready to use integrations for Apache and Nginx.

Some benefits of SEO.js to other approaches are:

- it's effortless, you don't need to setup and operate your own phantomjs server

- snapshots are created and cached in advance so the search engine crawler won't be put off by slow page loads

- snapshots are updated regularly

Re: Javascript apps can be fully crawlable

#107

Post by dsl »

cygx wrote: post #96
Volpe wrote: post #62

Earlier quoted context omitted.

... Comments have really gone to shit here haven't they. Some how we all end up antagonistic over bullshit like whether google have a big enough computer. But alas, you're right, google could never crawl with an actual browser - what a ridiculous suggestion. I apoligise for such a dumb-witted comment. As an aside: For my part in contributing such bad quality comments, I apoligise.

The point is that Google probably doesn't have a lot of cycles to spare - anything else wouldn't be good business sense. Anything that significantly adds to the load will lose them money - whether or not the operation needs to be realtime is secondary to that. I apologise for giving offense: I wrote the comment the same way I would have made it face-to-face, which is always a bit risky in a purely textual medium.

I don't know if you are trying to be serious at this point or not. Google has millions (literally) of machines with dozens of cores each. Search is their business that makes all the money.

Google executes JavaScript and renders the full DOM for every page internally. They generate full length screenshots of every page and have pointers to where text appears on the page so they can do highlighting of phrases within the screenshot.

It isn't even a debatable question if Google reuses the Chrome engine to do this.

Re: Javascript apps can be fully crawlable

#108

Post by ilaksh »

_lex wrote: post #8

This will get you penalized for having a website that takes forever to load. This is what happens: Googlebot requests page -> your webapp detects googlebot -> you call remote service and request that they crawl your website -> they request the page from you -> you return the regular page, with js that modifies it's look and feel -> the remote service returns the final html and css to your webapp -> your webapp return…

Its not a remote service. Its PhantomJS which is webkit rendering on your own server. Where did they say it was going to call a remote service?

Re: Javascript apps can be fully crawlable

#109

Post by Isofarro »

fuddle wrote: post #104

PhantomJS can be a pain to setup, I think the approach taken by Discourse.org is the best option: http://eviltrout.com/2013/06/19/adding-support-for-search-en...

Progressive enhancement is still better than this. The noscript element is a fallback when JavaScript is not available or turned off.

It doesn't handle situations where JavaScript is enabled, but your application failed to get the JavaScript completely to the browser.

With modern JavaScript and feature detection, the use of no script elements is a code smell.

Re: Javascript apps can be fully crawlable

#110

Post by eonil »

FedRegister wrote: post #71
eonil wrote: post #58

Static rendering of dynamic content? I don't think this does make sense. If it's pre-rednered, it's missing something. If it has all the data at first, then it's not dynamic. Pre-rendered(static) javascript app(dynamic)...? Hmm... I don't see anything more than something like JWT in JS instead of Java?

>Static rendering of dynamic content? I don't think this does make sense. Bro do you even Web 1.0? That's what CGI scripts in Perl did! Pull the data from the database, generate HTML (no JavaScript back then!) on the fly, and send to the browser.

JS is definitely client-side dynamic technology. At least from AJAX era.

Well... I don't understand how you and many people (including the author) can read JS as server-side dynamic in this HTML5 era...!!!

Post reply on HN