Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

21–30 of 171 posts

Re: Linux Pipes Are Slow

#21

Post by Lockal »

JoshTriplett wrote: post #14

This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…

This is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster.

All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch...

And they are not final, i. e. Noah Goldstein still updates them every year.

Re: Linux Pipes Are Slow

#22

Post by koverstreet »

JoshTriplett wrote: post #14

This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…

I'm still waiting for rep movsb and rep stosb to be fast enough to delete my simple C loop versions, for short memcpys.

Re: Linux Pipes Are Slow

#23

Post by jeffbee »

Lockal wrote: post #21
JoshTriplett wrote: post #14

This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…

This is not the full truth, "rep movsb" is fast until another threshold, after which either normal or non-temporal store is faster. All thresholds are described in https://codebrowser.dev/glibc/glibc/sysdeps/x86_64/multiarch... And they are not final, i. e. Noah Goldstein still updates them every year.

Which is these is "faster" depends greatly on whether you have the very rare memcpy-only workload, or if your program actually does something useful. Many people believe, often with good evidence, that the most important thing is for memcpy to occupy as few instruction cache lines as is practical, instead of being something that branches all over kilobytes of machine code. For comparison, see the x86 implementations in LLVM libc.

https://github.com/llvm/llvm-project/blob/main/libc/src/stri...

Re: Linux Pipes Are Slow

#24

Post by marcosdumay »

nitwit005 wrote: post #8

Just about every form of IPC is "slow". You have decided to pay a performance cost for safety.

You shouldn't have to pay that much. Pipes give you almost nothing, so they should cost almost nothing.

Specifically, there aren't many reasons for your fastest IPC to be slower than a long function call.

Re: Linux Pipes Are Slow

#25

Post by AkBKukU »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

Re: Linux Pipes Are Slow

#26

Post by effie »

hnlmorg wrote: post #5
henearkr wrote: post #3

Earlier quoted context omitted.

Pipes are extremely useful. But I guess it just depends on your use case. I do a lot of scripting. If you dislike their (relative) slowness, it's open source, you can participate in making them faster. And I'm sure that after this HN post we'll see some patches and merge requests.

The very thing that makes pipes useful is what also makes them slow. I don't think there is much we can do to fix that without breaking POSIX compatibility entirely. Personally I think there's much worse ugliness in POSIX than pipes. For example, I've just spent the last couple of days debugging a number of bugs in a shell's job control code (`fg`, `bg`, `jobs`, etc). But despite its warts, I'm still grateful we have…

What possible bugs can there be in those? They are quite simple to use and work as expected.

Re: Linux Pipes Are Slow

#27

Post by Izkata »

RevEng wrote: post #11

I didn't quite grasp why the original splice has to be so slow. They pointed out what made it slower than vmsplice - in particular allocating buffers and using scalar instructions - but why is this necessary? Why couldn't splice just be reimplemented as vmsplice? I'm sure there is a good reason, but I've missed it.

> Why couldn't splice just be reimplemented as vmsplice?

A possible answer that's currently just below your comment: https://news.ycombinator.com/item?id=41351870

> vmslice doesn't work with every type of file descriptor.

Re: Linux Pipes Are Slow

#28

Post by whartung »

AkBKukU wrote: post #25
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

What about domain sockets?

It's clumsier, to be sure, but if performance is your goal, the socket should be faster.

Re: Linux Pipes Are Slow

#29

Post by AkBKukU »

whartung wrote: post #28
AkBKukU wrote: post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

What about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.

It looks like FFmpeg does support reading from sockets natively[1], I didn't know that. That might be a better solution in this case, I'll have to look into some C code for writing my output to a socket to try that some time.

[1] https://ffmpeg.org/ffmpeg-protocols.html#unix

Re: Linux Pipes Are Slow

#30

Post by nitwit005 »

marcosdumay wrote: post #24
nitwit005 wrote: post #8

Just about every form of IPC is "slow". You have decided to pay a performance cost for safety.

You shouldn't have to pay that much. Pipes give you almost nothing, so they should cost almost nothing. Specifically, there aren't many reasons for your fastest IPC to be slower than a long function call.

If you don't think pipes offer much, don't use them.

Saying "long function call" doesn't mean much since a function can take infinitely long.

Post reply on HN