Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

51–60 of 171 posts

Re: Linux Pipes Are Slow

#51

Post by billfruit »

Ultimatt wrote: post #44
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

A better analogy is its like a society that uses steam trains attempting to industrially compete with a society that uses bullet trains (literally similar by factor of improvement). The UK built its last steam train for national use in 1960, four years later the Shinkansen was in use in Japan. Which of those two nations has a strong international industrial base in 2024?

Well the Mallard's top speed was very close to the first generation Shikansen 0 trains.

Re: Linux Pipes Are Slow

#52

Post by mort96 »

foota wrote: post #19
wakawaka28 wrote: post #16

Earlier quoted context omitted.

Buffering is there for a reason and this approach will lead to weird failure modes and fragility in scripts. The core issue is that any stream producer might go slower than any given consumer. Even a momentary hiccup will totally mess up the pipe unless there is adequate buffering, and the amount needed is system-dependent.

Maybe I misunderstand, but if the ring buffer is full isn't it ok for the sender to just block?

Yeah, and if the ring buffer is empty it's okay for the receiver to just block... exactly as happens today with pipes

Re: Linux Pipes Are Slow

#53

Post by teo_zero »

duped wrote: post #47
teo_zero wrote: post #38

Earlier quoted context omitted.

Genuine question: why does printf need a retry loop when using pipes?

It doesn't that's why no one does it. But for pipes what it means is that if whoever is reading or writing the pipe expects non blocking semantics, the other end needs to agree. And if they don't you'll eventually get an error because the reader or writer outpaced the other, and almost no program handles errors for stdin or stdout.

But even writing to a file doesn't guarantee non-blocking semantics. I still don't get what is special about pipes.

Re: Linux Pipes Are Slow

#54

Post by Borg3 »

Haha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow!

Anyway, nice article, its good to know whats going on under the hood.

Re: Linux Pipes Are Slow

#56

Post by ptx »

whartung wrote: post #28
AkBKukU wrote: post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

What about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.

Why should sockets be faster?

Re: Linux Pipes Are Slow

#57

Post by tyingq »

MaxBarraclough wrote: post #55
Borg3 wrote: post #54

Haha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow! Anyway, nice article, its good to know whats going on under the hood.

I'd assumed Cygwin pipes are just Windows pipes, is that not the case?

Not a comprehensive list of problems, and not current but a good illustrative post of the kind of issues that people have run into in this post:

https://cygwin.com/pipermail/cygwin-patches/2016q1/008301.ht...

Re: Linux Pipes Are Slow

#58

Post by Sesse__ »

AkBKukU wrote: post #25
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

At some point, I had a similar issue (though not related to licensing), and it turned out it was faster to do a high-bitrate H.264-encode of the stream before sending it over the FFmpeg socket than sending the raw RGBA data, even over localhost… (There was some minimal quality loss, of course, but it was completely irrelevant in the big picture.)

Re: Linux Pipes Are Slow

#60

Post by messe »

koverstreet wrote: post #12

One of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)

> and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer

Is there planned to be a standardized way to signal to the other end of the pipe that ring buffers are supported, so this could be handled transparently in libc? If not, I don't really see what advantage it gets you compared to shared memory + a futex for synchronization—for pipes that is.

Post reply on HN