Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

Post reply on HN
Print view View original

51–60 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#51

Post by winternewt »

Too wrote: post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Re: How fast are Linux pipes anyway? (2022)

#52

Post by [deleted] »

DiabloD3 wrote: post #12

TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…

[deleted]

Re: How fast are Linux pipes anyway? (2022)

#53

Post by eru »

4death4 wrote: post #22
mg wrote: post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

That may have been surprising, but, if you think about it a little deeper, it makes perfect sense. Programs in a pipeline execute concurrently. If they didn’t, pipelines wouldn’t be useful. For instance a pipeline that downloads a tar file with curl and then untars it. If you wait for curl to finish before running tar, you run in to all sorts of problems. For instance, where do you store the intermediate tar file if…

> If they didn’t, pipelines wouldn’t be useful.

Pipes would still be a useful way to structure your program. They would just be less useful.

Re: How fast are Linux pipes anyway? (2022)

#54

Post by thequux »

rixed wrote: post #50
jstimpfle wrote: post #5

Earlier quoted context omitted.

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

I don't think your message (or others) does justice to the original blogpost. Yes the pipe runs two subcommands in parallel but that is not why the blogpost is interesting (or its author surprised). It's because 'echo red' is supposed to block , thus introducing synchronization between the two branches of the pipe, yet it doesn't! And I must confess, when reading the command my first though was: "Ok so that first ech…

The pipe isn't broken, though; at least not until the second echo terminates. The kernel doesn't know that echo will never read stdin, because echo is generally a very simple program that doesn't bother closing unused file descriptors. Instead, the pipe is broken when there's nothing with an open receiving end, i.e., when the rightmost echo process terminates. Until then, it's just like any other pipe

Re: How fast are Linux pipes anyway? (2022)

#55

Post by dietrichepp »

rixed wrote: post #50
jstimpfle wrote: post #5

Earlier quoted context omitted.

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

I don't think your message (or others) does justice to the original blogpost. Yes the pipe runs two subcommands in parallel but that is not why the blogpost is interesting (or its author surprised). It's because 'echo red' is supposed to block , thus introducing synchronization between the two branches of the pipe, yet it doesn't! And I must confess, when reading the command my first though was: "Ok so that first ech…

> It's because 'echo red' is supposed to block,

It is not actually supposed to block. Pipes block when they are full, but there's not enough data here to fill a pipe buffer. When pipes are broken, SIGPIPE is sent to the writer. Pipes do not block just because nobody is reading from the read end--as long as the read end is still open somewhere, a process could read from it, and that is enough.

When you see "blue", what happened is the left-hand side of the pipe got killed because the right-hand side already finished before "echo red", which closed the read end completely, and then "echo red" got killed with SIGPIPE. That takes out "echo green" with it, because "echo" is a built-in, and so "echo" is not a subprocess. If you use "/bin/echo red" instead, then "green" will always be printed (because SIGPIPE is going to /bin/echo, and not the entire shell).

In other circumstances, the "echo blue" will never read stdin, but the kernel doesn't know or care. As far as the kernel is concerned, "echo blue" could possibly read from stdin, as long as stdin is open.

Re: How fast are Linux pipes anyway? (2022)

#56

Post by db48x »

winternewt wrote: post #51
Too wrote: post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Not necessarily. Good comments go a long way.

Re: How fast are Linux pipes anyway? (2022)

#57

Post by nesarkvechnep »

winternewt wrote: post #51
Too wrote: post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

In the end they'll just use a Lambda.

Re: How fast are Linux pipes anyway? (2022)

#58

Post by dietrichepp »

jcrites wrote: post #48
tuatoru wrote: post #35

Earlier quoted context omitted.

Thank you for taking the time to write this very detailed and lucid explanation.

For additional clarification, `echo` doesn’t read from stdin, so `… | echo xyz` doesn’t do what you probably assume. Try running `echo a | echo b` and you’ll see that only “b” is printed. That’s because `echo b` doesn’t read the “a” sent to it on stdin (and also doesn’t print it). If you want a program to read from stdin and write to stdout, you can use the `cat`, e.g. `echo a | cat` will print “a”. Lastly, be aware…

The way that shell builtins behave differently here is that SIGPIPE can take out the whole shell on the left side when echo is built-in.

When you /bin/echo red, then it's a subprocess, and its parent shell continues on, so you always get green somewhere in the output.

Re: How fast are Linux pipes anyway? (2022)

#59

Post by lelanthran »

packetlost wrote: post #15
DiabloD3 wrote: post #12

TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…

This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.

The problems with pipes is that passing a message involves a kernel context switch, no matter how small the message is.

Passing a message in-process is orders of magnitude faster than passing a message out-of-process.

Re: How fast are Linux pipes anyway? (2022)

#60

Post by loondri »

This article talks about making Linux pipes faster, but other methods like shared memory or message queues might still be quicker. For example, in systems that need to move a lot of data quickly, the extra steps with pipes could slow things down. Also, when many threads are sharing data, pipes might cause more problems than other methods. So, the improvements in the article might not help much in real-world situations where speed is crucial.
Post reply on HN