Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I want to write Node: Stream all the things

caolanmcmahon.com

Post reply on HN
Print view View original

61–70 of 118 posts

Re: How I want to write Node: Stream all the things

#61

Post by gruseom »

nailer wrote: post #31
glenjamin wrote: post #6

If you're doing Node.js, Caolan's async library is pretty much part of the standard toolkit. I know Caolan's been thinking about this and reworking it for a while, so I'll be interested to see whether it manages to see significant takeup.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

I have a question about the async library: what does it do if an asynchronous function or a callback raises an exception? The word "catch" doesn't appear in https://github.com/caolan/async/blob/master/lib/async.js.

I ask because I wrote some Lisp macros (I work in a Lisp that compiles to JS) to implement a few async patterns I need, and making sure that exceptions are trapped and threaded into the callback chain correctly was the most complicated part.

Re: How I want to write Node: Stream all the things

#62

Post by ixmatus »

aegiso wrote: post #52
ixmatus wrote: post #3

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO. Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its found…

Here's the main enlightenment of becoming a node.js guy: Node follows the unix way. Everything is a stream. It's just Buffers and JS objects flying around. It's really stupid, and sometimes it's nasty. This isn't helped by Javascript's warts. But there's an enormous upside to this: following the stupid Unix way means that no matter what you need to do with your data, there's an npm module for it. Just .pipe() your st…

I don't know node very well and only Javascript minimally but I understand what streaming I/O is and have used generators / iteratees for a long time in many languages.

I understand the benefits and that's why Pipes in HS is such an exciting thing because it gives us a formally reasoned and general set of stream computing tools - you can compute anything with type-level guarantees. It's just as flexible and general as, say, Unix pipes but better because there are guarantees of the library's tooling and there are guarantees of the programs you produce! You can't say that in Node / JS, Python, Ruby, etc...

JS's lack of strong typing limits your ability to reason about streams (a lot more than just streams, too) and further limits your ability to write performant stream computing software. Pipes, in Haskell, give you the big three: Effects (I/O), Composition (function composition with fusion), and Streams (generators and iteratees); because of the type system Haskell (and some nudges here and there by the library author) can fuse and optimize that code to a ridiculous degree in addition to all of the other nice guarantees you get from the type system (separate of I/O from pure code, etc...)

I personally don't think dynamic typing is a selling point, ever - I write software faster and with fewer bugs in Haskell than I ever have before in Python, Ruby, Erlang, or Scheme. But that's a totally different topic and I don't want to derail this one.

Don't misunderstand me as being aggressive, please. I fully respect what people decide to like and work on, I'm just trying to expand the awareness that there are tools in existence that do it better.

Re: How I want to write Node: Stream all the things

#63

Post by ixmatus »

jkrems wrote: post #35
ixmatus wrote: post #3

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO. Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its found…

Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.

Right, but the burden is on the programmer to ensure those types. If the stream composition is of simple things, like:

    cat some.txt | sort | uniq > yay.txt
Then it isn't a problem - it's obvious and simple, but I think there will be difficulties as the programs get larger and type-level awareness occupies more space in the programmer's brain vs. it being handled by the compiler...

Re: How I want to write Node: Stream all the things

#64

Post by caolanmcmahon »

caolanmcmahon wrote: post #20
eplawless wrote: post #17

I'm curious which features highland provides which RxJS doesn't. From what I understand, composable streams from any data source with backpressure support is pretty much the definition of Rx. Sometimes simplicity is a feature, too, though.

Rx doesn't handle back-pressure or laziness, so it's for only really for handling events.

RxJS advocates are unhappy with this comment so I'm going to qualify it a little. Apologies for any misunderstanding...

Rx doesn't handle automatic back-pressure (like Node Streams) but does have mechanisms to avoid overwhelming slow consumers. Rx also has delayed subscription which you can call lazy, but not by turning the stream into a pull-stream (allowing you to sequence actions in the way Highland does).

If any of the above needs further qualification or comment please weigh in on the issue by commenting here... but for now I'll leave it at that. I actually list RxJS in the blogpost because it's a good example!

Re: How I want to write Node: Stream all the things

#65

Post by mattpodwysocki »

caolanmcmahon wrote: post #64
caolanmcmahon wrote: post #20

Earlier quoted context omitted.

Rx doesn't handle back-pressure or laziness, so it's for only really for handling events.

RxJS advocates are unhappy with this comment so I'm going to qualify it a little. Apologies for any misunderstanding... Rx doesn't handle automatic back-pressure (like Node Streams) but does have mechanisms to avoid overwhelming slow consumers. Rx also has delayed subscription which you can call lazy, but not by turning the stream into a pull-stream (allowing you to sequence actions in the way Highland does). If any…

Coming in 2.3, we will have full capabilities for backpressure. We already have window/buffer/throttle, etc. But, I think it's naive to have only one style of backpressure because many are valid. Just an example of RxJS, and what can be done, which includes a style in which you can do several forms of backpressure, and yes, push to pull based models: https://gist.github.com/mattpodwysocki/9010149

Still fleshing it out, but pretty close to calling it complete: https://github.com/Reactive-Extensions/RxJS/tree/master/src/...

We're more than open to pull requests though if anyone thinks we're missing something here.

Re: How I want to write Node: Stream all the things

#66

Post by delluminatus »

gruseom wrote: post #61
nailer wrote: post #31

Earlier quoted context omitted.

From Great British Node Conf, maybe two months ago: a speaker talking about promises asked what people use for flow control: - promises: about 20% of the room. - async: about 80% of the room For me async.waterfall([list of functions]) is little nicer than 10 chained .thens(). And people advocating promises still keep saying it keeps things flat. No it doesn't, we're already flat because we're all using async. Stop pr…

I have a question about the async library: what does it do if an asynchronous function or a callback raises an exception? The word "catch" doesn't appear in https://github.com/caolan/async/blob/master/lib/async.js . I ask because I wrote some Lisp macros (I work in a Lisp that compiles to JS) to implement a few async patterns I need, and making sure that exceptions are trapped and threaded into the callback chain cor…

Last I checked, exceptions were not widely used in javascript because the try... catch block was a huge performance loss. I forget why exactly -- I want to say that the browser would spin up a whole new interpreter for catch blocks, just like with eval -- and it might be fixed in more modern JS engines, but I've still never seen exceptions used in JS. So I wouldn't be surprised if async doesn't handle them at all.

Re: How I want to write Node: Stream all the things

#67

Post by zmanian »

ixmatus wrote: post #3

Forgive me for being "That Guy" but I really think Javascript is ill-suited for this paradigm! Streams, honestly, are hard to keep straight when the program gets big without a stronger type system. IMHO. Some really sharp people have been working on stream computing software in Haskell for a while - Gabriel's Pipes package is a good example of generalized stream computing with strong equational reasoning as its found…

I'm a Haskeller but I'll defend Node a bit.

The absolute most compelling thing about Node is how it has hacked organizational dynamics in large companies. Walmart and PayPal basically used it to completely liberate their frontend groups from their backend systems using facade system with huge improvements on customer systems.

What is sad is that all the other high concurrency systems are going to end up implementing much of the GHC runtime without the reliability of Haskell...

Re: How I want to write Node: Stream all the things

#68

Post by jkrems »

ixmatus wrote: post #63
jkrems wrote: post #35

Earlier quoted context omitted.

Streams are being used in node.js already with a lot of success. Both basic byte chunk streams and object streams. As long as components are well behaved and only emit uniform data (everything they emit is of the same type), it's not better or worse than any function call.

Right, but the burden is on the programmer to ensure those types. If the stream composition is of simple things, like: cat some.txt | sort | uniq > yay.txt Then it isn't a problem - it's obvious and simple, but I think there will be difficulties as the programs get larger and type-level awareness occupies more space in the programmer's brain vs. it being handled by the compiler...

I didn't say loose/dynamic typing would not put the burden on the programmer to ensure the types. And it definitely has it's problems. I just don't think that this is specific to streams, it's true for every kind of composition.

Re: How I want to write Node: Stream all the things

#70

Post by aegiso »

ixmatus wrote: post #62
aegiso wrote: post #52

Earlier quoted context omitted.

Here's the main enlightenment of becoming a node.js guy: Node follows the unix way. Everything is a stream. It's just Buffers and JS objects flying around. It's really stupid, and sometimes it's nasty. This isn't helped by Javascript's warts. But there's an enormous upside to this: following the stupid Unix way means that no matter what you need to do with your data, there's an npm module for it. Just .pipe() your st…

I don't know node very well and only Javascript minimally but I understand what streaming I/O is and have used generators / iteratees for a long time in many languages. I understand the benefits and that's why Pipes in HS is such an exciting thing because it gives us a formally reasoned and general set of stream computing tools - you can compute anything with type-level guarantees . It's just as flexible and general…

> JS's lack of strong typing limits your ability to reason about streams (a lot more than just streams, too) and further limits your ability to write performant stream computing software

I think you missed my point so I'll restate: in practice you don't reason about streams in Node, because the community (a product of the simplicity of the streams API) has a packaged solution to your problem. It plugs right in. And this ecosystem exists because of the simplicity and dynamicity of the constructs used.

I actually agree with you that Haskell does it "better". It's purer and cleaner. You'll probably have less bugs if you write everything in Haskell.

Except it doesn't matter to me, because Haskell doesn't have anything close to the plug-and-playability of npm modules -- and this is a pure social product of the stupid interface that Node exposes compared to Haskell. Node is shittier, and that's why it's more capable at solving the problem I have -- constructing powerful apps in close to no time, and zero lines of my own code.

I guess what I'm saying is that sometimes worse is better.

Post reply on HN