Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

71–80 of 171 posts

Re: Linux Pipes Are Slow

#71

Post by immibis »

messe wrote: post #60
koverstreet wrote: post #12

One of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)

> and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer Is there planned to be a standardized way to signal to the other end of the pipe that ring buffers are supported, so this could be handled transparently in libc? If not, I don't really see what advantage it gets you compared to shared memory + a futex for synchronization—for pipes that is.

Presumably the same interface still works if the other side is using read/write.

Re: Linux Pipes Are Slow

#72

Post by quietbritishjim »

CyberDildonics wrote: post #32
AkBKukU wrote: post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

The parent comment mentioned license incompatibility, which I guess would still apply if they used ffmpeg as a library.

Re: Linux Pipes Are Slow

#73

Post by fatcunt »

> I do not know why the JMP is not just a RET, however.

This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk.

https://github.com/torvalds/linux/blob/v6.1/arch/x86/include...

https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret...

> The NOP instructions at the beginning and at the end of the function allow ftrace to insert tracing instructions when needed.

These are from the ASM_CLAC and ASM_STAC macros, which make space for the CLAC and STAC instructions (both of them three bytes in length, same as the number of NOPs) to be filled in at runtime if X86_FEATURE_SMAP is detected.

https://github.com/torvalds/linux/blob/v6.1/arch/x86/include...

https://github.com/torvalds/linux/blob/v6.1/arch/x86/include...

https://github.com/torvalds/linux/blob/v6.1/arch/x86/kernel/...

Re: Linux Pipes Are Slow

#76

Post by samastur »

Someone wrote: post #40
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works.

If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

Re: Linux Pipes Are Slow

#77

Post by jiehong »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Replace “Linux pipes” by “Electron apps”, and people would not agree.

Also, why leave performance on the table by default? Just because “it should be enough for most people I can think of”?

Add Tesla motors to a Toyota Corolla and now you’ve got a sportier car by default.

Re: Linux Pipes Are Slow

#78

Post by [deleted] »

samastur wrote: post #76
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

[deleted]

Re: Linux Pipes Are Slow

#79

Post by paulannesley »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Sometimes the best answer really is a faster Corolla!

https://www.toyota.com/grcorolla/

(These machines have amazing engineering and performance, and their entire existence is a hack to work around rules making it unviable to bring the intended GR Yaris to the US market.. Maybe just enough eng/perf/hack/market relevance to HN folk to warrant my lighthearted reply. Also, the company president is still on the tools.

Re: Linux Pipes Are Slow

#80

Post by crabbone »

ploxiln wrote: post #45
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Indeed. In the author's case, the slow pipe is moving data at 17 GB/s which is over 130 gbps. I've used pipes for a lot of stuff over 10+ years, and never noticed being limited by the speed of the pipe, I'm almost certain to be limited by tar, gzip, find, grep, nc ... (even though these also tend to be pretty fast for what they do).

I had two cases in my practice where pipes were slow. Both related to developing a filesystem.

1. Logging. At first our tools for reading the logs from a filesystem management program were using pipes, but they would be overwhelmed quickly (even before it would overwhelm pagers and further down the line). We had to write our own pager and give up on using pipes.

2. Storage again, but a different problem: we had a setup where we deployed SPDK to manage the iSCSI frontend duties, and our component to manage the actual storage process. It was very important that the communication between these two components be as fast and as memory-efficient as possible. The slowness of pipes comes also from the fact that they have to copy memory. We had to extend SPDK to make it communicate with our component through shared memory instead.

So, yeah, pipes are unlikely to be the bottleneck of many applications, but definitely not all.

Post reply on HN