Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

91–100 of 171 posts

Re: Linux Pipes Are Slow

#91

Post by caf »

duped wrote: post #47
teo_zero wrote: post #38

Earlier quoted context omitted.

Genuine question: why does printf need a retry loop when using pipes?

It doesn't that's why no one does it. But for pipes what it means is that if whoever is reading or writing the pipe expects non blocking semantics, the other end needs to agree. And if they don't you'll eventually get an error because the reader or writer outpaced the other, and almost no program handles errors for stdin or stdout.

Making the read side non-blocking doesn't affect the write side, and vice-versa.

Re: Linux Pipes Are Slow

#92

Post by duped »

caf wrote: post #91
duped wrote: post #47

Earlier quoted context omitted.

It doesn't that's why no one does it. But for pipes what it means is that if whoever is reading or writing the pipe expects non blocking semantics, the other end needs to agree. And if they don't you'll eventually get an error because the reader or writer outpaced the other, and almost no program handles errors for stdin or stdout.

Making the read side non-blocking doesn't affect the write side, and vice-versa.

That is not true for pipes.

Re: Linux Pipes Are Slow

#93

Post by rwmj »

qsantos wrote: post #86
rwmj wrote: post #85

Be interesting to see a version using io_uring, which I think would let you pre-share buffers with the kernel avoiding some copies, and avoid syscall overhead (though the latter seems negligible here).

That sounds like a good idea!

I'm not claiming it'll be faster! Additionally io_uring has its own set of challenges, such as whether it's better to allocate one ring per core or one ring per application (shared by some or all cores). Pre-sharing buffers has trade-offs too, particularly in application complexity [alignment, you have to be careful not to reuse a buffer before it is consumed] versus the efficiency of zero copy.

Re: Linux Pipes Are Slow

#94

Post by caf »

koverstreet wrote: post #12

One of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)

Presumably ringbuffer_wait() can also be signalled through making it 'readable' in poll()?

Re: Linux Pipes Are Slow

#95

Post by caf »

duped wrote: post #92
caf wrote: post #91

Earlier quoted context omitted.

Making the read side non-blocking doesn't affect the write side, and vice-versa.

That is not true for pipes.

It is, at least on Linux for ordinary pipe(2) pipes.

I just wrote up a test to be sure: in the process with the read side, set it to non-blocking with fcntl(p, F_SETFL, O_NONBLOCK) then go to sleep for a long period. Dump a bunch of data into the writing side with the other process: the write() call blocks once the pipe is full as you would expect.

Re: Linux Pipes Are Slow

#96

Post by Sesse__ »

jraph wrote: post #65
Sesse__ wrote: post #58

Earlier quoted context omitted.

At some point, I had a similar issue (though not related to licensing), and it turned out it was faster to do a high-bitrate H.264-encode of the stream before sending it over the FFmpeg socket than sending the raw RGBA data, even over localhost… (There was some minimal quality loss, of course, but it was completely irrelevant in the big picture.)

> There was some minimal quality loss, of course, but it was completely irrelevant in the big picture But then the solutions are not comparable anymore, are they? Would a lossless codec instead have improved speed?

No, because I had hardware H.264 encoder support. :-) (The decoding in FFmpeg on the other side was still software. But it was seemingly much cheaper to do a H.264 software decode.)

Re: Linux Pipes Are Slow

#97

Post by Borg3 »

MaxBarraclough wrote: post #55
Borg3 wrote: post #54

Haha. When I read the title I smiled. Linux pipes slow? Moook.. Now try Cygwin pipes. Thats what I call slow! Anyway, nice article, its good to know whats going on under the hood.

I'd assumed Cygwin pipes are just Windows pipes, is that not the case?

Its not that easy. Yeah, they are, but there is a lot of POSIX like glue inside so they work correctly with select() and other alarms. Code is very complicated.

But still, kudos for Cygwin Developers for creating Cygwin :) Great work, even tho it have some issues.

Re: Linux Pipes Are Slow

#98

Post by whiterknight »

CyberDildonics wrote: post #32
AkBKukU wrote: post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

And you go from having a well defined modular interface that’s flexible at runtime to a binary dependency.

Re: Linux Pipes Are Slow

#99

Post by whiterknight »

samastur wrote: post #76
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

You’re describing the outcome of one individual person. Money is just a tool for allocating resources. Saving 1 million of resources is a good thing.

Re: Linux Pipes Are Slow

#100

Post by donaldihunter »

Something I didn't see mentioned in the article about AVX512, aside from the xsave/xrstor overhead, is that AVX512 is power hungry and causes CPU frequency scaling. See [1], [2] for details and as an example of how nuanced it can get.

[1] https://www.intel.com/content/dam/www/central-libraries/us/e...

[2] https://www.intel.com/content/www/us/en/developer/articles/t...

Post reply on HN