Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

41–50 of 171 posts

Re: Linux Pipes Are Slow

#41

Post by adrian_b »

koverstreet wrote: post #22
JoshTriplett wrote: post #14

This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…

I'm still waiting for rep movsb and rep stosb to be fast enough to delete my simple C loop versions, for short memcpys.

It is likely that on recent CPUs they are always faster than C loop versions.

On my Zen 3 CPU, for lengths of 2 kB or smaller it is possible to copy faster than with "rep movsb", but by using SIMD instructions (or equivalently the builtin "memcpy" provided by most C compilers), not with a C loop (unless the compiler recognizes the C loop and replaces it with the builtin memcpy, which is what some compilers will do at high optimization levels).

Re: Linux Pipes Are Slow

#44

Post by Ultimatt »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

A better analogy is its like a society that uses steam trains attempting to industrially compete with a society that uses bullet trains (literally similar by factor of improvement). The UK built its last steam train for national use in 1960, four years later the Shinkansen was in use in Japan. Which of those two nations has a strong international industrial base in 2024?

Re: Linux Pipes Are Slow

#45

Post by ploxiln »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Indeed. In the author's case, the slow pipe is moving data at 17 GB/s which is over 130 gbps.

I've used pipes for a lot of stuff over 10+ years, and never noticed being limited by the speed of the pipe, I'm almost certain to be limited by tar, gzip, find, grep, nc ... (even though these also tend to be pretty fast for what they do).

Re: Linux Pipes Are Slow

#46

Post by [deleted] »

adrian_b wrote: post #39
Lockal wrote: post #21

Earlier quoted context omitted.

This is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster. All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch... And they are not final, i. e. Noah Goldstein still updates them every year.

It depends on the CPU. There is no good reason for "rep movsb" to be slower at any big enough data size. On a Zen 3 CPU, "rep movsb" becomes faster than or the same as anything else above a length slightly greater than 2 kB. However there is a range of multi-megabyte lengths, which correspond roughly with sizes below the L3 cache but exceeding the L2 cache, where for some weird reason "rep movsb" becomes slower than…

[deleted]

Re: Linux Pipes Are Slow

#47

Post by duped »

teo_zero wrote: post #38
duped wrote: post #36

Earlier quoted context omitted.

I agree with this but with a much more nuanced take: avoid pipes if either reader or writer expects to do async i/o and you don't own both the reader and writer. In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the l…

Genuine question: why does printf need a retry loop when using pipes?

It doesn't that's why no one does it.

But for pipes what it means is that if whoever is reading or writing the pipe expects non blocking semantics, the other end needs to agree. And if they don't you'll eventually get an error because the reader or writer outpaced the other, and almost no program handles errors for stdin or stdout.

Re: Linux Pipes Are Slow

#48

Post by qsantos »

I am again getting the hug of death of Hacker News. The situation is better than the last time thanks to caching WordPress pages, but loading the page can still take a few seconds, so bear with me!

Re: Linux Pipes Are Slow

#49

Post by ploxiln »

AkBKukU wrote: post #25
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

What percentage of CPU time is used by the pipe in this scenario? If pipes were 10x faster, would you really notice any difference in wall-clock-time or overall-cpu-usage, while this decoding SDK is generating the raw data and ffmpeg is processing it? Are these video processing steps anywhere near memory copy speeds?

Re: Linux Pipes Are Slow

#50

Post by khafra »

effie wrote: post #26
hnlmorg wrote: post #5

Earlier quoted context omitted.

The very thing that makes pipes useful is what also makes them slow. I don't think there is much we can do to fix that without breaking POSIX compatibility entirely. Personally I think there's much worse ugliness in POSIX than pipes. For example, I've just spent the last couple of days debugging a number of bugs in a shell's job control code (`fg`, `bg`, `jobs`, etc). But despite its warts, I'm still grateful we have…

What possible bugs can there be in those? They are quite simple to use and work as expected.

They work as expected on Redhat and Debian. "POSIX" leaves open a lot of possibility for less-well-tested systems. They could be writing shellscripts on Minix or HelenOS.
Post reply on HN