Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Effect of Pipe Capacity on Unix Pipeline Performance

dongyuxuan.me

Post reply on HN
Print view View original

11–17 of 17 posts

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#11

Post by cat199 »

jandrese wrote: post #7
thedance wrote: post #6

I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.

The vast majority of shell script writers don't know or care what the pipe size is. The tools they are using treat them as just one solid stream of data. There's not much to break.

the shell isn't the only user of pipe(2) - that said, since you can't count on fixed scheduling on either end, most things likely will handle things okay

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#12

Post by duskwuff »

> If the two processes exchange data in this pattern [one-to-one exchange], a small pipe doesn’t cause unnecessary blockings.

I don't think this is correct! Context switches are expensive -- a small pipe size will force both processes to make more system calls to move the same amount of data through the pipe.

In this light, the UNIX specification's requirement of 4 KB is definitely too small; even the Linux 2.6+ default of 64 KB feels like it might be on the small side.

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#13

Post by jschwartzi »

thedance wrote: post #8
jandrese wrote: post #7

Earlier quoted context omitted.

The vast majority of shell script writers don't know or care what the pipe size is. The tools they are using treat them as just one solid stream of data. There's not much to break.

I meant in the kernel. Like if I hooked up a fuzzer and just started swinging the pipe size around.

Considering there's a semi-contiguous allocation in the kernel I bet you'd find some interesting bugs that way. I don't think very many people mess with the pipe sizes other than to tune to a multiple of the input/output block size for the application. And if you're trying to maximize throughput you're likely to reach your optimum block count long before you start blowing out buffers.

Actually, come to think of it named pipes would be another good place to fuzz.

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#14

Post by pushedx »

anonsivalley652 wrote: post #5

Awhhh. :'( I wanted to see: - What happens when piping an 8 TiB file through awk to be stored on another physical volume (proly want to use xfs). - A graph of various block sizes, various file sizes, pipeline sizes and timing using a HPET. I don't know how you make an 2+-way pipe like it the diagrams except by process substitution, except that doesn't always work and you can't create cycles without creating FIFO (nam…

The diagrams are not diagrams of pipelines, they are diagrams of a single pipeline `a | b`, over time, with points in time shown when a is writing and when b is reading.

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#15

Post by duskwuff »

thedance wrote: post #6

I wonder how much stuff breaks if you actually change the pipe size. I bet there’s a lot of assumptions kicking around.

Very little. The Linux default changed from 4 KB to 64 KB in 2.6.11, and the macOS XNU kernel will actually adjust pipe sizes dynamically based on the size of writes and system memory pressure.

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#16

Post by dongyx »

steerablesafe wrote: post #2

I expected some examples and benchmarks with some common unix tools.

You're right. I didn't provide enough examples. I found this theory in my real-world project. I've done a series of experiments and benchmarks. However, the project is commercial and I can't demonstrate it. I'll build some simple examples in the future if I have time.

Re: The Effect of Pipe Capacity on Unix Pipeline Performance

#17

Post by dongyx »

duskwuff wrote: post #12

> If the two processes exchange data in this pattern [one-to-one exchange], a small pipe doesn’t cause unnecessary blockings. I don't think this is correct! Context switches are expensive -- a small pipe size will force both processes to make more system calls to move the same amount of data through the pipe. In this light, the UNIX specification's requirement of 4 KB is definitely too small; even the Linux 2.6+ defa…

You're right. The words I took are not precise. I don't know how to express the concept that the performance is not essentially affected. Is there a universally acknowledged concept to distinguish the difference? Like the concept of time complexity in the algorithm area.
Post reply on HN