Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

Post reply on HN
Print view View original

31–40 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#31

Post by gpderetta »

jstimpfle wrote: post #20
packetlost wrote: post #15

Earlier quoted context omitted.

This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.

Pipes are FIFO data buffers implemented in the kernel. For communication between threads of the same process, you can replace any pipe object by a userspace queue implementation protected by e.g. mutex + condition variable. It is functionally equivalent and has potential to be faster. And if you wrap all accesses in lock/unlock pairs (without locking any other objects in between) there is no danger of introducing any…

Incidentally you can use the exact same setup (plush mmap) for interprocess queues.

The advantage of threads is that you can pass pointers to your data through the queue, while that's harder to do between processes and you have to resort to copying data in the queue instead.

Re: How fast are Linux pipes anyway? (2022)

#32

Post by dilyevsky »

gpderetta wrote: post #17
DiabloD3 wrote: post #12

TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…

Pipes are zero copy only if you use splice or vmsplice. These linux specific syscalls are hard to use (particularly wmsplice) and the vast majority of programs and shell filters (with the notable exception of pv) don't use them and pay for the cost of copying in and out of kernel memory.

If you’re using Go it will automatically splice your reader/writer when using io.Copy, etc

Re: How fast are Linux pipes anyway? (2022)

#33

Post by jeromegn »

jcrites wrote: post #30

Are there good data handling libraries that provide abstractions over pipes, sockets, files, and memory and implement optimizations like these? I'd be interested in knowing if there are such libraries in C, C++, Rust, or other systems languages. I wasn't familiar with some of the APIs mentioned in the article like splice() and vmsplice(), so I wondered if there are libraries that I might use when building ~low-level…

There’s a crate for tokio, so it’s not automatic but might still be interesting: https://lib.rs/crates/tokio-splice

Re: How fast are Linux pipes anyway? (2022)

#34

Post by ldoughty »

whalesalad wrote: post #27

Love the Edward Tuftian aesthetic of this site. Although above a certain viewport width I would imagine you want a `margin: 0 auto` to center the content block. On a 27" display it is tough to read without resizing the window.

I have to agree... I really like the side-notes to get more details/explanation. You can skip the side-notes to keep reading and stay on the main story, but get what normally would be included in parenthesis or otherwise as an in-line comment.... Best of both worlds here I think. If I actively maintained a blog, I'd probably steal this design! :-)

Re: How fast are Linux pipes anyway? (2022)

#35

Post by tuatoru »

jstimpfle wrote: post #5
mg wrote: post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

Thank you for taking the time to write this very detailed and lucid explanation.

Re: How fast are Linux pipes anyway? (2022)

#37

Post by duped »

jcrites wrote: post #30

Are there good data handling libraries that provide abstractions over pipes, sockets, files, and memory and implement optimizations like these? I'd be interested in knowing if there are such libraries in C, C++, Rust, or other systems languages. I wasn't familiar with some of the APIs mentioned in the article like splice() and vmsplice(), so I wondered if there are libraries that I might use when building ~low-level…

This may go against the grain but this isn't really worth abstracting over since it's not portable. You'll probably want to implement it by hand everywhere you need it.

Higher level code only uses them rarely because they're pretty special purpose and they have to be specialized for Linux. If you're shuffling data around without looking at it only on Linux, splice is useful. There's not that many applications that have that property (something like say, TCP/UDP proxies definitely need it - but your bog standard HTTP server? Not so much).

And if you are writing these apps then the buzzwords like "zero copy" come up often, and splice is one of the first results you'll see.

Re: How fast are Linux pipes anyway? (2022)

#38

Post by leodag »

Racing0461 wrote: post #26
mg wrote: post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

Chatgpt was able to figure this out with a simple "what does the following do". But it could also be a case of chatgpt being trained on your article. >>> Note: The ordering of "green" and "blue" in the output might vary because these streams (stdout and stderr) might be buffered differently by the shell or operating system. Most commonly, you will see the output as illustrated above.

That's wrong though, it's got nothing to do with different buffering (which is usually done at the application level, by the way).

Re: How fast are Linux pipes anyway? (2022)

#39

Post by emmelaich »

ldoughty wrote: post #34
whalesalad wrote: post #27

Love the Edward Tuftian aesthetic of this site. Although above a certain viewport width I would imagine you want a `margin: 0 auto` to center the content block. On a 27" display it is tough to read without resizing the window.

I have to agree... I really like the side-notes to get more details/explanation. You can skip the side-notes to keep reading and stay on the main story, but get what normally would be included in parenthesis or otherwise as an in-line comment.... Best of both worlds here I think. If I actively maintained a blog, I'd probably steal this design! :-)

Is there some standard css/html way of pushing side notes or pics into the first column if viewing width is too small?

That would be the best of both worlds!

Re: How fast are Linux pipes anyway? (2022)

#40

Post by djbusby »

packetlost wrote: post #15
DiabloD3 wrote: post #12

TL;DR: Maximum pipe speed, assuming both programs are written as optimally as possible, is approximately the speed of what one core in your system can read/write; this is because, essentially, the kernel maps the same physical memory page from one program's stdout to the other's stdin, thus making the operation a zerocopy (or a fast onecopy in slightly less optimal situations). I've known this one for awhile, and it…

This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.

Like how Postfix works. That's a fun architecture to look at. Multiple processes and file based queue. Meanwhile I panic if I don't have PostgreSQL to save my data :/
Post reply on HN