Still the main problems is not solved: you risk getting penalized for serving a different content to the googlebot
Javascript apps can be fully crawlable
51–60 of 112 posts
Re: Javascript apps can be fully crawlable
#52Earlier quoted context omitted.
* At scale, without massive performance drops
I'm confused, search indexing isn't a realtime exercise... Why would performance be an issue? Running a headless browser vs running "whatever it is they run that can execute JS" doesn't seem like a huge leap...
At Google's scale any performance drop can have massive implications. If Google's crawl rate is 100 million[1] pages a day then a 1% drop in crawl rate means 1 million less pages crawled per day (which has many implications, for example having to use more compute power to regain crawl rate which raises costs etc.)
You are right that it is not a real-time exercise but they do have crawl targets.
You cannot be flippant about "Why don't they just do X" when scale is that big.
[1] Picked out of the air but probably in the right magnitude (or even a little small)
Re: Javascript apps can be fully crawlable
#53This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…
That Google Ajax crawler spec is no magic bullet.
Nick Denton: "Dip in uniques largely because of drop in Google refers. Pageviews (which are driven more by core audience) less affected." -- http://twitter.com/nicknotned/status/61152134929981440
Nick Denton: "Google does not fully support "hashbang" URLs. So we're eliminating them rather than waiting for Mountain View." -- http://twitter.com/nicknotned/status/61465859079671808
Nick Denton: "Yeah, I'd advise against hashbang urls. Will kill search traffic -- even if you abide by Google protocol." -- http://twitter.com/nicknotned/status/62595141927583745
Re: Javascript apps can be fully crawlable
#54Earlier quoted context omitted.
Have you ever experienced web apps that laggs like crap? Yeah think about that x 10000 million web pages.
... right but a bot doesn't get impatient. So I don't see your point.
They should just shut down all their data centers and crawl the whole web from a single box located in someone's basement.
After all, the bot doesn't get impatient.
Re: Javascript apps can be fully crawlable
#55A better way is to do a hybrid single/multipage app as described here:
https://news.ycombinator.com/item?id=6507135
It's a multipage app, that uses ajax to function as a singlepage app. From the user's point of view it's a singlepage app, but it's accessible from any of the URLs that it pushStates to, so it's like the best of both worlds. It's fully crawlable because it functions as a multipage app, but it's got the speed of a singlepage app (if your browser supports pushState)
Re: Javascript apps can be fully crawlable
#56Making JS heavy sites crawlable is also possible with libraries like https://github.com/minddust/jquery-pjaxr and https://github.com/defunkt/jquery-pjax . Plus the push state has the advantage of "real" urls.
Re: Javascript apps can be fully crawlable
#57Very cool, but something I don't get:
- Try to go to prerender.io, press "Install It -> Ruby on rails". Now it loads the ruby on rails example.
- Then go all the way down and change to "Prerendereed content". Pressing "Install It -> Ruby on rails" doesn't do anything now.
Shouldn't it render the same content? "Add the middleware gem to your Gemfile..." and so on.
Re: Javascript apps can be fully crawlable
#58Static rendering of dynamic content? I don't think this does make sense.
If it's pre-rednered, it's missing something. If it has all the data at first, then it's not dynamic.
Pre-rendered(static) javascript app(dynamic)...? Hmm... I don't see anything more than something like JWT in JS instead of Java?
Re: Javascript apps can be fully crawlable
#59Static rendering of dynamic content? I don't think this does make sense. If it's pre-rednered, it's missing something. If it has all the data at first, then it's not dynamic. Pre-rendered(static) javascript app(dynamic)...? Hmm... I don't see anything more than something like JWT in JS instead of Java?
> Static rendering of dynamic content?
Yes.
> I don't think this does make sense.
It does, if you use one of the JS frameworks listed on the linked page.
Re: Javascript apps can be fully crawlable
#60If you are able to "pre-render" a JavaScript app like this, then you should be serving users the pre-rendered version and then enhancing it with JavaScript after onload. JavaScript-only apps are a blight on the web. All it takes is a bad SSL cert, or your CDN going down, and your pages become useless to the end-user.
All it takes is a bad SSL cert, or your CDN going down, and your pages become useless to the end-user.
How are non-JavaScript pages protected from this?