Node.js stream handbook
github.com
Node.js stream handbook
1–10 of 18 posts
Re: Node.js stream handbook
#2This is a great write-up on streams. For curiosity sake, I'd be interested to know the overhead of
var stream = fs.createReadStream(__dirname + '/data.txt');
stream.pipe(res);
and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.
Re: Node.js stream handbook
#3This is a great write-up on streams. For curiosity sake, I'd be interested to know the overhead of var stream = fs.createReadStream(__dirname + '/data.txt'); stream.pipe(res); vs. fs.readFile('./data.txt) and res(). and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.
> ... and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.
fs.readFile(...) reads the entire contents of the file into memory. For small files that may be acceptable but if you're dealing with anything that could be large it's not going to be very pleasant or even work properly.
I just tried it out on my laptop for a 100MB and a 1GB file. For a 100MB file using streams is about 25% slower. However the sync method failed for the 1GB file:
Style 100MB 1GB
===== ===== ===
fs.readFileSync .176s
streams/pipe .234s 1.25sRe: Node.js stream handbook
#4This is a great write-up on streams. For curiosity sake, I'd be interested to know the overhead of var stream = fs.createReadStream(__dirname + '/data.txt'); stream.pipe(res); vs. fs.readFile('./data.txt) and res(). and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case.
> ... and if streams become a better and better option as response sizes get larger and larger. The streams syntax is nicer looking in any case. fs.readFile(...) reads the entire contents of the file into memory. For small files that may be acceptable but if you're dealing with anything that could be large it's not going to be very pleasant or even work properly. I just tried it out on my laptop for a 100MB and a 1GB…
By adapting the highWaterMark option one could probably tweak it even further.
Default buffer size seems to be 64k: https://github.com/joyent/node/blob/912b5e05811fd24f09f9d652...
Stream buffering: http://www.nodejs.org/api/stream.html#stream_buffering
Re: Node.js stream handbook
#5Also by the magnificent substack, learning by doing done right: the node.js stream adventure!
Install via:
npm install -g stream-adventure
stream-adventure
Re: Node.js stream handbook
#6Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
Re: Node.js stream handbook
#7Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
Promises hep deal with asynchronous functions by providing a convenient way to pass success/fail methods in a way some people prefer more than classic callbacks.
Streams are a totally different use case. Streams are used when you want to process data one part at a time without having to have all of the data in memory at once.
*edit: typo
Re: Node.js stream handbook
#8Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
With streams you don't wait for them to end. They may never end, in fact.
Re: Node.js stream handbook
#9Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
Promises hep deal with asynchronous functions by providing a convenient way to pass success/fail methods in a way some people prefer more than classic callbacks. Streams are a totally different use case. Streams are used when you want to process data one part at a time without having to have all of the data in memory at once. *edit: typo
Promises are actually about trust, not syntax. With a callback you have to trust that the function that will invoke your callback (which might not be something you wrote) will only ever call it once, that it will pass through errors, that it won't call both your success and failure callbacks, etc.
With a promise (or rather, following the promise specification) you instead get back an object that you choose how to handle. That object is either pending or else an immutable success or failure result. Either success or failure will be called (not both), and whichever result is called can only be called once. As such, promises allow you to avoid inversion of control and to safely interact with potentially untrusted code.
This great series of articles helped me understand this: http://blog.getify.com/promises-part-2/
Re: Node.js stream handbook
#10Not hoping to start any sort of flame war, but could someone try to explain this difference between streams and promises? The conceptual difference doesn't seem completely clear to me.
You may be interested in going through "A General Theory of Reactivity"[0] from Kris Kowal, creator of the q promise library. It discusses relationships between streams, promises, iterators and generators (although the streams that he discusses are not how Node.js streams are implemented)