Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

41–50 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#41
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

I'm genuinely curious, how else could this work? It's like spawning threads, it's inherently indeterministic.

Re: How fast are Linux pipes anyway? (2022)

#42
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

tl;dr Piped commands run in parallel not in serial.

(The data "runs" in serial.)

Re: How fast are Linux pipes anyway? (2022)

#43

Earlier quoted context omitted.

I have to agree... I really like the side-notes to get more details/explanation. You can skip the side-notes to keep reading and stay on the main story, but get what normally would be included in parenthesis or otherwise as an in-line comment.... Best of both worlds here I think. If I actively maintained a blog, I'd probably steal this design! :-)

Is there some standard css/html way of pushing side notes or pics into the first column if viewing width is too small? That would be the best of both worlds!

responsive design concepts would enable this

Re: How fast are Linux pipes anyway? (2022)

#44
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

If one enjoys fast, 0-copy I/O on Linux, here's an article.[0]

PS: Precision of language to avoid confusion: "Indeterministic" is a philosophy term, while the CS term is "nondeterministic".

0. https://blog.superpat.com/zero-copy-in-linux-with-sendfile-a...

Re: How fast are Linux pipes anyway? (2022)

#48
post #35

Earlier quoted context omitted.

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

Thank you for taking the time to write this very detailed and lucid explanation.

For additional clarification, `echo` doesn’t read from stdin, so `… | echo xyz` doesn’t do what you probably assume. Try running `echo a | echo b` and you’ll see that only “b” is printed. That’s because `echo b` doesn’t read the “a” sent to it on stdin (and also doesn’t print it).

If you want a program to read from stdin and write to stdout, you can use the `cat`, e.g. `echo a | cat` will print “a”.

Lastly, be aware that `echo` is usually a shell builtin that functions like `print`. I’m not sure of all the ways that it might behave differently, but something to be aware of (that it’s not a child process like `cat`).

Re: How fast are Linux pipes anyway? (2022)

#49
So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time.

Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 20x slower than what is possible.

Exceptionally written article btw.

Re: How fast are Linux pipes anyway? (2022)

#50
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

I don't think your message (or others) does justice to the original blogpost.

Yes the pipe runs two subcommands in parallel but that is not why the blogpost is interesting (or its author surprised). It's because 'echo red' is supposed to block, thus introducing synchronization between the two branches of the pipe, yet it doesn't!

And I must confess, when reading the command my first though was: "Ok so that first echo will die with a SIGPIPE and stderr will be all about the broken pipe." And I was wrong, because of that small buffer.

I wonder what other unices do allow a write to a broken pipe to complete successfully?

Post reply on HN