Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I want to write Node: Stream all the things

caolanmcmahon.com

Post reply on HN
Print view View original

31–40 of 118 posts

Re: How I want to write Node: Stream all the things

#31

Post by nailer »

glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control:

- promises: about 20% of the room.

- async: about 80% of the room

For me async.waterfall([list of functions]) is little nicer than 10 chained .thens().

And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pretending async doesn't exist and isn't massively popular.

And way, way better documented. Q.spawn what? And this is the best promises library?

Stack Overflow question: Simplest fs.readFile example with generators and Q?

Current only answer:

  Q.spawn(function* () {
      …
      var data = yield Q.ninvoke(fs, "readFile", somefile);
      …
  });
Answer from Highland docs:

    var data = _.wrapCallback(fs.readFile)('myfile');
- What's Q (yes it's a module, but what does it mean? Is it supposed to a misspelt queue or something else?

- What does 'ninvoking' something do?

- Shouldn't I just be able to to put the variable declaration outside of the scope?

- Why do competing Open Source implementations of the same standard exist? Can't there just one reference implementation?

That's not the future.

I might be really ignorant here. I probably am - I could read a shit tonne of docs to work out what this strange beast does and technically someone can probably do a better job answering that Stack Overflow question. But nobody has, because very few people know how to operate the current state of the art generators/promises setup.

From the Q docs: "If you have a number of promise-producing functions that need to be run sequentially"

No, I don't have a number of promise producing functions. Nobody in nodeland has that. I just have functions. I could read about turning them into promise producing functions, and calculate whether this abstraction layer is adding value, but then again, I could do productive work with async.

And from the looks of it, Highland too.

Re: How I want to write Node: Stream all the things

#32

Post by hamburglar »

jashkenas wrote: post #28
Cthulhu_ wrote: post #26

Earlier quoted context omitted.

I agree; what I personally wonder is why Underscore and similar libraries don't make use of Javascript's prototype business and add methods to the array and object prototypes? Probably things I'm overlooking here, but, [1, 2, 3].map(stuff) is much nicer than _.map([1, 2, 3], stuff) and the like.

Ha ha. You're describing the state of affairs before Underscore existed, back when functional-ish programming in JavaScript was ruled by Prototype.js: http://prototypejs.org/ ... which added a lot of useful methods to native prototypes. While handy in controlled and limited environments, mucking about with native prototypes quickly becomes extremely dangerous and difficult — once you have two third-party modules on t…

Whereas with _.map, you always know exactly which implementation you're getting, right? :)

Re: How I want to write Node: Stream all the things

#33

Post by jonny_eh »

nailer wrote: post #31
glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

Bluebird is the best promises library: https://github.com/petkaantonov/bluebird

Re: How I want to write Node: Stream all the things

#34

Post by owenversteeg »

Cthulhu_ wrote: post #26
hamburglar wrote: post #10

This is highly intriguing, but I must ask why the JS community has this fascination with obscure identifiers like "_". It decreases readability when what should be a logically-chosen descriptive identifier for your class is replaced with a single character that visually recedes into the language syntax. Edit: I've grudgingly given jquery a pass on this because of its ubiquity, but come on, a stream library? Not to me…

I agree; what I personally wonder is why Underscore and similar libraries don't make use of Javascript's prototype business and add methods to the array and object prototypes? Probably things I'm overlooking here, but, [1, 2, 3].map(stuff) is much nicer than _.map([1, 2, 3], stuff) and the like.

Most people are very hesitant to add methods to array, string, etc. because if more than one person does that you'll end up with all sorts of problems.

Re: How I want to write Node: Stream all the things

#35

Post by jkrems »

ixmatus wrote: post #3

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO. Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its found…

Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.

Re: How I want to write Node: Stream all the things

#36

Post by wcummings »

al2o3cr wrote: post #23
wcummings wrote: post #21

Earlier quoted context omitted.

If you look at the full docs, there is a lot of overlap, highland provides a lot of of the functions from underscore.js, and since from what I can tell its trying to unify the "javascript utility belt", I can only imagine more will be added.

"highland provides a lot of of the functions from underscore.js" "A lot of the functions" (versus "all of the functions") sounds like a one-way ticket to readability hell, since it means that an inattentive reader may assume the functions ARE from Underscore.

To me the benefit of a convenient shorthand outweighs this, someone would figure it out really quick, seems like a non-issue. A few different libs use $ and the world doesn't end.

Re: How I want to write Node: Stream all the things

#37

Post by jkrems »

nailer wrote: post #31
glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

async:

    async.waterfall([
      fn1, fn2, fn3
    ], function(err, result) {
    });
Q:

    resultP = [ fn1, fn2, fn3 ].reduce(Q.when, void 0);
    resultP
    .then(function(result) {});
    .catch(function(err) {});
If you chain thens in Q, you are not doing it right (imho).

Re: How I want to write Node: Stream all the things

#38

Post by phleet »

nailer wrote: post #31
glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

Promises aren't just about keeping things flat.

The biggest value to me is being able to avoid the passing around of callbacks and them relying on varying conventions (some async are function(args, ..., callback(err, data)), some are function(args, ..., callback(data), errback(err)), some are function({success: callback, error: errback})).

Promises solve this problem by not passing around callbacks _at all_. Instead you return the promise and let the consumer attach the callback itself. And once we have promises widely available and part of the standard library, the calling conventions will be standardized too.

EDIT: I agree that as it stands, the lack of standardization of promises (jQuery's are mutable, for instance) is a pain, and that documentation could certainly be better.

Re: How I want to write Node: Stream all the things

#39

Post by alasdair_ »

glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

Agreed, almost everyone at Groupon is using Async over the alternatives.

Re: How I want to write Node: Stream all the things

#40

Post by spion »

nailer wrote: post #31
glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

  var Promise = require('bluebird');
  var fs = Promise.promisifyAll(require('fs'));

  Promise.spawn(function* () { 
    var data = yield fs.readFileAsync(somefile);
  });
Documentation: https://github.com/petkaantonov/bluebird/blob/master/API.md

My favorite example is doing a diff using an async diff service which provides a function `svc.diff(string1, string2)`. But also imagine that you need to preprocesses the files using the sync function `removeEmptyLines(buffer)`. This is how the function looks like when implemented using Bluebird:

  function diffTwoFiles(f1, f2) {
    var file1 = fs.readFileAsync(f1).then(removeEmptyLines),
        file2 = fs.readFileAsync(f2).then(removeEmptyLines);
    return Promise.join(file1, file2).spread(svc.diff);
  }
I'd love to see someone come up with a better example using callbacks and async.
Post reply on HN