Live data from Hacker News

How I want to write Node: Stream all the things

caolanmcmahon.com

111–118 of 118 posts

Re: How I want to write Node: Stream all the things

#111
post #90

Earlier quoted context omitted.

It doesn't do anything about thrown exceptions. The correct way to deal with an error in asynchronous code is to pass an object describing the error as the first argument to the callback. Any code that takes a callback is expected to know this and not throw exceptions. From a practical perspective, it doesn't make sense to try to catch exceptions in asynchronous code, anyway. Once you do something asynchronous, you l…

The correct way to deal with an error in asynchronous code is to pass an object describing the error as the first argument to the callback. Sure, but what if the error is thrown at you as an exception in the first place—which happens a fair amount, because that's how the JS runtime tells you when something is wrong? How do you get from there to the callback way? What the Lisp macro I mentioned does is generate a sepa…

> Sure, but what if the error is thrown at you as an exception in the first place—which happens a fair amount, because that's how the JS runtime tells you when something is wrong? How do you get from there to the callback way?

Its on you to catch that, not your libraries. This shouldn't be terribly common, though. The only thing I can remember having to wrap in a try/catch in the codebase I work on is JSON.parse.

> The async library could do the same, albeit with a lot more code. I'm curious why it doesn't.

It couldn't, without domains. try-catch wouldn't do it. Domains are something that is not very well understood, in my experience, and expected to happen at a higher level than libraries like async.

Re: How I want to write Node: Stream all the things

#112
post #90

Earlier quoted context omitted.

The correct way to deal with an error in asynchronous code is to pass an object describing the error as the first argument to the callback. Sure, but what if the error is thrown at you as an exception in the first place—which happens a fair amount, because that's how the JS runtime tells you when something is wrong? How do you get from there to the callback way? What the Lisp macro I mentioned does is generate a sepa…

> Sure, but what if the error is thrown at you as an exception in the first place—which happens a fair amount, because that's how the JS runtime tells you when something is wrong? How do you get from there to the callback way? Its on you to catch that, not your libraries. This shouldn't be terribly common, though. The only thing I can remember having to wrap in a try/catch in the codebase I work on is JSON.parse. > T…

I don't understand most of this. For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur (mostly its calls to the functions that get passed in to it). It would be interesting if it couldn't, since then we'd have an example of something macros can do that functions cannot. But it seems obvious to me that it could; you'd just need a lot of try-catches. What am I missing?

As for domains, I don't know what you mean by them, but if they're catching errors at a higher level than the async library, my guess is that they must be some more sophisticated sort of top-level handler; perhaps something that keeps track of which async calls are in progress and attempts to bind exceptions back to their context? Whatever it is, it sounds complicated.

But what I understand least of all is how you guys all seem to write Javascript code that generates almost no exceptions. To me that sounds almost like bug-free code. No null references, for example? I get stuff like that all the time.

Re: How I want to write Node: Stream all the things

#113

Earlier quoted context omitted.

> Sure, but what if the error is thrown at you as an exception in the first place—which happens a fair amount, because that's how the JS runtime tells you when something is wrong? How do you get from there to the callback way? Its on you to catch that, not your libraries. This shouldn't be terribly common, though. The only thing I can remember having to wrap in a try/catch in the codebase I work on is JSON.parse. > T…

I don't understand most of this. For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur (mostly its calls to the functions that get passed in to it). It would be interesting if it couldn't, since then we'd have an example of something macros can do that functions cannot. But it seems obvious to me that it could; you'd just need a lot of try-catches. W…

> No null references, for example?

We write in CoffeeScript, where a null reference check is so astonishingly easy to write that you use them everywhere you might get a null. I'm not sure what other exceptions you're seeing. We do basically no math, so /0 errors aren't a problem.

> For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur

Let's build a typical function you might pass to async:

function(next) { request.get(url, function(err, data) { JSON.parse(data); } }

Let's assume the server doesn't serve JSON like we expect - so JSON.parse throws an exception. The only thing async could have wrapped in a try/catch is the main function, but we've fired off a request and then the call stack wrapped up, including the try/catch. Next, an event occurs that calls our callbacks, not going through async at all. That's where the exception occurs. The stack trace generated by that exception doesn't contain any code in the async lib, so it can't possibly have a try/catch active.

Domains are a way of fixing this. You create a Domain and bind callbacks to it - if that callback throws an exception, the Domain instead emits an error event.

Re: How I want to write Node: Stream all the things

#114

Earlier quoted context omitted.

I don't understand most of this. For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur (mostly its calls to the functions that get passed in to it). It would be interesting if it couldn't, since then we'd have an example of something macros can do that functions cannot. But it seems obvious to me that it could; you'd just need a lot of try-catches. W…

> No null references, for example? We write in CoffeeScript, where a null reference check is so astonishingly easy to write that you use them everywhere you might get a null. I'm not sure what other exceptions you're seeing. We do basically no math, so /0 errors aren't a problem. > For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur Let's build a t…

Ok, thanks, I get it now. In my case a macro transforms the body of each callback to catch exceptions and pass them back as error args through the callback chain. So in your example, there would be a generated try-catch around the JSON.parse(data). I forgot this detail (sign of a successful abstraction?) and it does seem an example of something macros can do that functions cannot.

Re null reference checks, to get behavior analogous to a null exception you have not only to check for null, but also pass back an explicit error if you find it. That's a lot more work than adding in an extra question mark. Null checks that do nothing but not crash are a mixed blessing; 90+% of the time they do what you want, but when they don't, you get a silent failure and a debugging goose chase. I'd be surprised if you told me that that never happens.

I took a look at Node.js domains and they do seem really complicated. If I were working in Javascript instead of having control over the language, I doubt I would use them; I would probably just crash-and-restart as one of the other commenters described. That's not a good solution, but probably the best tradeoff given the alternatives.

Re: How I want to write Node: Stream all the things

#115

Earlier quoted context omitted.

> No null references, for example? We write in CoffeeScript, where a null reference check is so astonishingly easy to write that you use them everywhere you might get a null. I'm not sure what other exceptions you're seeing. We do basically no math, so /0 errors aren't a problem. > For example, I don't know why you say that the async library couldn't try-catch every place that an exception might occur Let's build a t…

Ok, thanks, I get it now. In my case a macro transforms the body of each callback to catch exceptions and pass them back as error args through the callback chain. So in your example, there would be a generated try-catch around the JSON.parse(data). I forgot this detail (sign of a successful abstraction?) and it does seem an example of something macros can do that functions cannot. Re null reference checks, to get beh…

Our use-case for domains is to allow the process to finish serving its other in-progress reqs before crashing. When an error occurs, we stop accepting new connections in that process, give them 10-15 seconds to complete, and then do the crash-and-restart cycle.

That said, we get very thorough testing from our large user base, and we quickly fix crashers. Our server proc crash rate is almost 0, brought up by occasional spikes on releases.

Re: How I want to write Node: Stream all the things

#116
post #75
post #47

Earlier quoted context omitted.

There is nothing wrong with Q, but Bluebird is a bit more node-oriented and also has really, really low CPU/memory overhead (lower than caolan's async). Also it provides the best debugging experience, period - because of its long stack traces spanning multiple previous async events. I didn't quite understand the comment about data being a scope down. What do you mean? Yes, promises do have quite a steep learning curv…

Thanks for the awesome blog post. I read it extensively when I was trying to use promises for everything. I ended up deciding that it was not worth the trouble of learning a lib with like 30 methods for a tiny bit of syntax sugar. Callbacks have never even really bothered me. Named functions FTW.

Those methods are there for convenience. Most of the time while I'm working with promises, I don't use anything else except `Promise.all` and `Promise.prototype.then`. Similarly how to when working with caolan's async, most of the time you don't use anything else except waterfall, series, parallel, mapSeries and map. (Note however that async's utility grab bag approach results with a larger commonly used subset :P)

Promises are not about syntax sugar. They're about utilizing the whole power of the language and providing a parallel for most features found in synchronous code:

1. Functions have return values

When using node style callbacks, we're ignoring the fact that the language was designed with functions that have return values. Instead we use half-functions. Its no wonder those compose quite badly - the language wasn't designed for that kind of composition. The language was designed to work with functions that take input values and return an output value. Callback-based functions do only the first half. Thats why to get them to compose we resort to a bunch of hairy helpers and boilerplate code.

Callback-based functions that don't return anything are seriously crippled in power, and promises fix that, restoring much of the power.

2. Errors can bubble like exceptions

When using node style callbacks, we must explicitly handle all errors. On one hand, this is a good thing: we should deal with all errors. On the other hand, its quite tedious: most of the time we can't deal with the error at the exact place it appears but must pass it up one level in the call chain.

Promises do the error bubbling automatically. We can attach the appropriate error handler at the appropriate place to deal with the error.

This simple feature results with tons of useful patterns, one of which is the ability to manage resources with constructs like C#'s `using` keyword. - https://github.com/spion/promise-using

3. Values in variables can be accessed multiple times

When using node style callback and event emitters, we must make sure to "capture" the value exactly when it comes. If we don't do that, poof, its gone forever - we missed it.

In contrast, promises will keep the value for us. If we need to access that value later, we can simply attach another callback handler. An example where this may be useful is a database connection:

We initialize the connection and get a promise for that connection:

  var pConn = db.connect(host, port);
How do we implement a query method that is immediately available and will queue up queries until the connection is established? Easily:

  function query(q, params) {
    return pConn.then(function(conn) {
      return conn.queryAsync(q, params);
    });
  }
It doesn't matter whether the connection was established a long time ago or hasn't been established yet - the query will either execute immediately or its execution will be delayed until the connection becomes available.

Now try doing this with callbacks :)

Re: How I want to write Node: Stream all the things

#117

Earlier quoted context omitted.

Whereas with _.map, you always know exactly which implementation you're getting, right? :)

Actually, you do. You've loaded it, and you can lock it down privately to your library or app with _.noConflict(). You can have ten different versions of Underscore loaded on the page, living in peace and harmony, in ten different third-party modules. Not that you should. But that you could.

Extending native prototypes creates other less obvious hurdles for libraries too. Craft.js, es5-shim, Modernizr, MooTools, Prototype.js, and Sugar.js, to name a few, have all, at one time or another, added incorrect shims to native prototypes.

While Underscore is in a better position than those that extend native prototypes, regarding api/environment conflicts, it can still be tripped up by poor shims because it defers to many ES5 methods if they exist. For example, if Prototype 1.6.0 and Underscore.js are included on a page Underscore's `_.reduce` method won't work properly. This is one of the reasons why libs/frameworks like Dojo, Ember, Lo-Dash, RequireJS, Sizzle, and YUI do native checks too.

Re: How I want to write Node: Stream all the things

#118

Earlier quoted context omitted.

Native Promises already landed Chrome 32 and Q still does not support native promises. Bluebird delegate to native if supported. Promises can be used as flow control, but more importantly, it's an object that encapsulates asynchronous mechanics. I like to see how async can launch an asynchronous operation, then allow listeners to be attached later to capture the result. Now, you may say that if you want to attach lis…

This is false. Bluebird does not and never will delegate to native promises.

ok... then can you explain what's happening here: https://github.com/petkaantonov/bluebird/blob/master/js/brow...

Where the line checks for `window.Promise`

Post reply on HN