The Effect of Pipe Capacity on Unix Pipeline Performance
1–10 of 17 posts
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#2I expected some examples and benchmarks with some common unix tools.
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#3Believe it or not: I have to deal with issues just like this over in webdev-land! (See the Terminal app in https://dev.lotw.xyz/desk.os or skip right to it, go to https://dev.lotw.xyz/shell.os)
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#4I expected some examples and benchmarks with some common unix tools.
Ditto. This article feels like it was published before the author had finished vetting out everything he/she wanted to demonstrate.
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#5Awhhh. :'(
I wanted to see:
- What happens when piping an 8 TiB file through awk to be stored on another physical volume (proly want to use xfs).
- A graph of various block sizes, various file sizes, pipeline sizes and timing using a HPET. I don't know how you make an 2+-way pipe like it the diagrams except by process substitution, except that doesn't always work and you can't create cycles without creating FIFO (named pipes):
cmd0 ~joe/foo &
cmd4 (cmd6 here ~joe/foo)
(...) # outer can r/w
(...) # outer can wRe: The Effect of Pipe Capacity on Unix Pipeline Performance
#6I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#7I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.
The vast majority of shell script writers don't know or care what the pipe size is. The tools they are using treat them as just one solid stream of data. There's not much to break.
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#8I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.
The vast majority of shell script writers don't know or care what the pipe size is. The tools they are using treat them as just one solid stream of data. There's not much to break.
I meant in the kernel. Like if I hooked up a fuzzer and just started swinging the pipe size around.
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#9I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.
Unless there's a dialog going on, it shouldn't matter a great deal. But it's easy to mess up a dialog (having a conversation with both stdin and stdout).
Re: The Effect of Pipe Capacity on Unix Pipeline Performance
#10Awhhh. :'( I wanted to see: - What happens when piping an 8 TiB file through awk to be stored on another physical volume (proly want to use xfs). - A graph of various block sizes, various file sizes, pipeline sizes and timing using a HPET. I don't know how you make an 2+-way pipe like it the diagrams except by process substitution, except that doesn't always work and you can't create cycles without creating FIFO (nam…
Most shells don't have great support for making even non-linear graphs of (anonymous) pipe connections (though note that wilder, even dynamic graphs are possible with the "coproc" builtin of e.g. ksh, zsh, or modern bash).
However, arbitrary graphs are possible with the system calls available -- just create all the pipes in the spawning process with pipe(), and after fork(), but before exec() move the right ends into the file descriptors 0, 1, and 2 with dup2().