31–40 of 171 posts
Earlier quoted context omitted.
You shouldn't have to pay that much. Pipes give you almost nothing, so they should cost almost nothing. Specifically, there aren't many reasons for your fastest IPC to be slower than a long function call.
If you don't think pipes offer much, don't use them. Saying "long function call" doesn't mean much since a function can take infinitely long.
Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.
Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.
Just about every form of IPC is "slow". You have decided to pay a performance cost for safety.
Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.
Just about every form of IPC is "slow". You have decided to pay a performance cost for safety.
Pipes don’t exist for safety, they exist as an optimization to pass data between existing programs.
just never use pipes. they are some weird archaism that need to die :P the only time ive used them is external constraints. they are just not useful.
In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the last time you checked the error code of `printf` and put it in a retry loop?
just never use pipes. they are some weird archaism that need to die :P the only time ive used them is external constraints. they are just not useful.
I agree with this but with a much more nuanced take: avoid pipes if either reader or writer expects to do async i/o and you don't own both the reader and writer. In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the l…
This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…
This is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster. All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch... And they are not final, i. e. Noah Goldstein still updates them every year.
On a Zen 3 CPU, "rep movsb" becomes faster than or the same as anything else above a length slightly greater than 2 kB.
However there is a range of multi-megabyte lengths, which correspond roughly with sizes below the L3 cache but exceeding the L2 cache, where for some weird reason "rep movsb" becomes slower than SIMD non-temporal stores.
At lengths exceeding the L3 size, "rep movsb" becomes again the fastest copy method.
The Intel CPUs have different behaviors.
Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.
Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.