Live data from Hacker News

Javascript apps can be fully crawlable

prerender.io

91–100 of 112 posts

Re: Javascript apps can be fully crawlable

#91

Earlier quoted context omitted.

Progressive enhancement. It is a web development best practice. You will find that "dynamic loading of content" doesn't automatically mean "no content served by HTML under any circumstances". This is an error perpetuated by these JavaScript-only frameworks. For example, bustle.com - there is absolutely no customer experience reason for the website not to have the content loaded at the HTML layer and then progressivel…

"The content here isn't tied exclusively to the behaviour layer." Can you elaborate on a situation where that the content and the behaviour are tied together, and what you would do in that case? From my understanding, Facebook's BigPipe loads content in modules to reduce user perceived latency. If I'm building X site and wanted that same behavior (since I've heard on several occasions that there is a direct correlati…

Facebook's big pipe is nothing more than a client-side hack to work around a limitation of their server-side architecture.

Both Yahoo and Amazon - that I personally know of - have an infrastructure where components on the page are rendered separately and in parallel, and are stitched together on the HTML layer. The render time is then down to the rendering time of the slowest component, or the slowest dependency chain of components.

Loading content in with JavaScript after the HTML page load is always going to be slower, and perceivably so - look at both Twitter and AirBnb, both have written about how much faster they get content to the user using progressive enhancement.

If you decide that the HTML layer isn't the right layer for content, you are working against the strengths of the Web. And of course, that leads down a path where you are sacrificing SEO, sacrificing robustness.

Your time is better spent figuring out why it takes your server too long to generate content, and put in steps to reduce the server side labour.

The JavaScript include approach isn't quicker. Bustle.com for example, takes 10 seconds to show the first page - that's horrific.

Re: Javascript apps can be fully crawlable

#92

This is a great approach, but detecting the user-agent is the wrong way to decide if you should pre-render the page. If you include the following meta tag in the header: then Google will request the page with the "_escaped_fragment_" query param. That's when you should serve the pre-rendered version of the page. Google has documentation on this here: https://developers.google.com/webmasters/ajax-crawling/docs/... and…

no it is not a great approach for 99% of the cases.

the issue with getting content from scripted sites is not the initial part... you could use noscript and be done much easier.

the real issue is that most sites require user interaction to get to most content. this does nothing besides providing a convenient DoS entry point.

nice hack though.

Re: Javascript apps can be fully crawlable

#93
post #34
post #27

Earlier quoted context omitted.

Google actually recommend you do this, provided it's the same content that is shown with JS enabled: https://support.google.com/webmasters/answer/66353 "JavaScript: Place the same content from the JavaScript in a tag. If you use this method, ensure the contents are exactly the same as what’s contained in the JavaScript, and that this content is shown to visitors who do not have JavaScript enabled in their browser." T…

Placing the text content on the same page in a tag is entirely different than rendering different content based on user agent. That's what the noscript tag is meant to do -- Google is telling you to follow best practices for fallback to non-JS browsers.

and much easier btw

Re: Javascript apps can be fully crawlable

#94

Earlier quoted context omitted.

"The content here isn't tied exclusively to the behaviour layer." Can you elaborate on a situation where that the content and the behaviour are tied together, and what you would do in that case? From my understanding, Facebook's BigPipe loads content in modules to reduce user perceived latency. If I'm building X site and wanted that same behavior (since I've heard on several occasions that there is a direct correlati…

Facebook's big pipe is nothing more than a client-side hack to work around a limitation of their server-side architecture. Both Yahoo and Amazon - that I personally know of - have an infrastructure where components on the page are rendered separately and in parallel, and are stitched together on the HTML layer. The render time is then down to the rendering time of the slowest component, or the slowest dependency chai…

Okay, I'm sold on HTML being where the content should be on page load. My next question is are there any frameworks that can assist with this stitching together of content? I'm afraid my ignorance is showing here, but I can't off the top of my head list any.

The web paradigm that I've grown up with is the single-threaded dynamic content generation one, most recently using MVC, but with any server-side logic. The concept of parallel rendering of content and a "stitching" together of HTML is new to me.

I'm also curious as to the best practices surrounding page linking when the behavior specifies something like no screen flash. It seems like all that content (like, the whole page) would have to be loaded with AJAX.. then you're right back where you started with loading content with JS. Maybe it's forgivable as long as the initial page load returns a complete set of content?

Perhaps there is a place where I can learn more about the logistics of progressive enhancement.

Thanks for being willing to answer my questions about this. It's something I've always wanted hashed out in my head from an opponent of these frameworks.

Re: Javascript apps can be fully crawlable

#96
post #62
post #54

Earlier quoted context omitted.

They should just shut down all their data centers and crawl the whole web from a single box located in someone's basement. After all, the bot doesn't get impatient.

... Comments have really gone to shit here haven't they. Some how we all end up antagonistic over bullshit like whether google have a big enough computer. But alas, you're right, google could never crawl with an actual browser - what a ridiculous suggestion. I apoligise for such a dumb-witted comment. As an aside: For my part in contributing such bad quality comments, I apoligise.

The point is that Google probably doesn't have a lot of cycles to spare - anything else wouldn't be good business sense.

Anything that significantly adds to the load will lose them money - whether or not the operation needs to be realtime is secondary to that.

I apologise for giving offense: I wrote the comment the same way I would have made it face-to-face, which is always a bit risky in a purely textual medium.

Re: Javascript apps can be fully crawlable

#97

I recently needed to do this for google, but i wanted the rendering time, and delivery of the page to be under 500MS, so i hacked up something that works with expressjs https://github.com/icodeforlove/node-express-renderer It uses phantomjs but removes all the styles initially so the rendering time is much faster. (my ember app was averaging 70MS to render, but i prefetch the page data)

came across this recently and it's super easy to implement

Re: Javascript apps can be fully crawlable

#98

Earlier quoted context omitted.

Facebook's big pipe is nothing more than a client-side hack to work around a limitation of their server-side architecture. Both Yahoo and Amazon - that I personally know of - have an infrastructure where components on the page are rendered separately and in parallel, and are stitched together on the HTML layer. The render time is then down to the rendering time of the slowest component, or the slowest dependency chai…

Okay, I'm sold on HTML being where the content should be on page load. My next question is are there any frameworks that can assist with this stitching together of content? I'm afraid my ignorance is showing here, but I can't off the top of my head list any. The web paradigm that I've grown up with is the single-threaded dynamic content generation one, most recently using MVC, but with any server-side logic. The conc…

"are there any frameworks that can assist with this stitching together of content"

Mainstream, no. These are not typical use-cases for sites until they reach a gigantic scale.

Plus, even before you get to that level, there's heaps you can do on caching at different levels, pre-calculating, pre-generating. So many optimisations at various levels of your stack, then there's scaling across hardware. Wordpress and Wikipedia don't need paralellisation of HTML components yet.

"The concept of parallel rendering of content and a "stitching" together of HTML is new to me."

It was new to me until I joined Yahoo. The approach is breaking down a page into modular/independent sections. And then running some sort of parelellisation process, and when all the responses are received, then render the page with those generated components.

There's probably a variety of hacks in each major platform that will allow things to be parelellised. If you want to parallelise on the HTTP level, then curl multiget is an option: http://php.net/manual/en/function.curl-multi-init.php

It's probably possible to cobble together something with node.js too. Node.js receives the HTTP request, turns that into a series of service calls that return HTML, makes those calls asynchronously, waits for them all to return (this is where Node.js excels), then renders an HTML page skeleton, replacing placeholders with the responses from the various services. With a decent promise library that waits for a number of calls to finish, this is quite a compact approach, I guess.

Almost anything that allows asynchronous operations that uses resources outside of the current request handler can be fashioned into a component parallelisation stack.

"I'm also curious as to the best practices surrounding page linking when the behavior specifies something like no screen flash."

No screen flash is impossible, due to the nature of the Web. The browser has to receive the HTML first before it can know what dependent resources are needed. The problem you are trying to solve here is to minimise the perceived time between the HTML arriving, and enough of the CSS to load in for an incremental render to paint a close-enough-to-look-complete rendering.

Loading the content after the CSS is one way of doing that. Which replaces the screen-flash delay with a blank screen. That's the JavaScript-app approach.

I don't like that, because it delays the appearance of content.

The perception of screenflash can be minimised, mostly by decreasing the amount of traffic crossing the wire until a good enough repaint can happen. There are various tricks and hacks for minimising this, but due to the nature of the web they cannot be completely eliminated using Web technologies. They can be replaced with other issues.

Tricks I'd consider is reducing the amount of CSS needed to render the page, break the CSS up into a primary rendering and a secondary, more detailed rendering. The primary rendering is just a basic layout plus main elements styling. Perhaps a careful inline style or two, judicious display:nones and overflow: hiddens to minimise page assets moving around as incremental CSS rendering happens. Also, if you want to get serious, techniques for deferring CSS, JavaScript and images of content outside the current viewport is an option. Yahoo loved deferring the loading of avatar images in an article comments area till after onload. I see that technique used in tech publication websites, can't remember off the top of my head a site that did this.

"Perhaps there is a place where I can learn more about the logistics of progressive enhancement."

The process is about thinking about a site one layer at a time. Get it functional at each level: HTML with links and form posts, CSS presentational level, JavaScript enhancements and usability improvements. Like building a skyscraper, you get the foundation right first.

But before that, it takes understanding as to what are the core use-cases for the site. This is about tasks a visitor can complete. Something that's tied into key product indicators and metrics. I doubt bustle.com use page loading performance as their primary business success factor. It is more likely to be about customer activity - how long did they visit, how many articles, any social activity.

It's figuring out the primary services and functionality of the site, and building that to not rely on JavaScript, or CSS. Primary services are those that, if you didn't provide them, you'd have no business.

Secondary functionality and use-cases -- those that complement or are related to primary functionality, that can be argued on a case by case basis whether a quick ajax solution is sufficient. But most of the time when you get progressive enhancement right, it becomes just a natural development technique.

Gov.uk have a good explanation of progressive enhancement here: https://www.gov.uk/service-manual/making-software/progressiv...

Re: Javascript apps can be fully crawlable

#99
post #29
post #12

Earlier quoted context omitted.

> Google doesn't like it when they are shown different content than a browsing user. This is exactly correct. Regardless of your motivations.

Not a technically different page, specifically different content . Serving different pages to Google is fine, as long as they contain the same primary content that the real pages do. That's the whole point - so you can serve prerendered pages to Google but still have a JS-based frontend for the actual users.

AJAX sites often lazy load in content later. My point is the page delivered initially is not the same as the static version content wise or technically.

Re: Javascript apps can be fully crawlable

#100
post #18

I tried using phantomjs in the past to serverside render a complex backbone application for SEO, and it was taking over 15 seconds to return a response (which is bad for SEO). Looking at the prerender's source I did't see any caching mechanism. What kind of load times have you see rendering your apps? Have there been recent significant improvements in phantomjs's performance?

I run http://www.brombone.com. We provide prerendered snapshots as a service.

You can get it faster than 15 seconds, but you can't really get it fast enough. We precache everything. I would strongly recommend against trying to process the pages in realtime.

Post reply on HN