Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

61–70 of 171 posts

Re: Linux Pipes Are Slow

#62
post #26
post #5

Earlier quoted context omitted.

The very thing that makes pipes useful is what also makes them slow. I don't think there is much we can do to fix that without breaking POSIX compatibility entirely. Personally I think there's much worse ugliness in POSIX than pipes. For example, I've just spent the last couple of days debugging a number of bugs in a shell's job control code (`fg`, `bg`, `jobs`, etc). But despite its warts, I'm still grateful we have…

What possible bugs can there be in those? They are quite simple to use and work as expected.

I'm talking about shell implementation not shell usage.

To implement job control, there are several signals you need to be aware of:

- SIGSTSP (what the TTY sends if it receives ^Z)

- SIGSTOP (what a shell sends to a process to suspend it)

- SIGCONT (what a shell sends to a process to resume it)

- SIGCHLD (what the shell needs to listen for to see there is a change in state for a child process -- this is also sometimes referred to as SIGCLD)

- SIGTIN (received if a process read from stdin)

- SIGTOU (received if a process cannot write to stdout nor set its modes)

Some of these signals are received by the shell, some are by the process. Some are sent from the shell and others from the kernel.

SIGCHLD isn't just raised for when a child process goes into suspend, it can be raised for a few different changes of state. So if you receive SIGCHLD you then need to inspect your children (of course you don't know what child has triggered SIGCHLD because signals don't contain metadata) to see if any of them have changed their state in any way. Which is "fun"....

And all of this only works if you manage to fork your children with special flags to set their PGID (not PID, another meta ID which represents what process group they belong to), and send magic syscalls to keep passing ownership of the TTY (if you don't tell the kernel which process owns the TTY, ie is in the foreground, then either your child process and/or your shell will crash due to permission issues).

None of this is 100% portable (see footnote [1]) and all of this also depends on well behaving applications not catching signals themselves and doing something funky with them.

The bug I've got is that Helix editor is one of those applications doing something non-standard with SIGTSTP and assuming anything that breaks as a result is a parent process which doesn't support job control. Except my shell does support job control and still crashes as a result of Helix's non-standard implementation.

In fairness to Helix, my shell does also implement job control in a non-standard way because I wanted to add some wrappers around signals and TTYs to make the terminal experience a little more comfortable than it is with POSIX-compliant shells like Bash. But because job control (and signals and TTYs in general) are so archaic, the result is that there are always going to be edge case bugs with applications (like Helix) that have implemented things a little differently themselves too.

So they're definitely not easy to use and can break in unexpected ways if even just one application doesn't implement things in expected ways.

[1] By the way, this is all ignoring subtle problems that different implementations of PTYs (eg terminal emulators, terminal multiplexors, etc) and different POSIX kernels can introduce too. And those can be a nightmare to track down and debug!

Re: Linux Pipes Are Slow

#64
> I do not know why the JMP is not just a RET, however.

The jump seems generated by the expansion of the `ASM_CLAC` macro, which is supposed to change the EFLAGS register ([1], [2]). However in this case the expansion looks like it does nothing (maybe because of the target ?). I 'd be interested to know more about that. Call to the wild.

[1]: https://github.com/torvalds/linux/blob/master/arch/x86/inclu...

[2]: https://stackoverflow.com/a/60579385

Re: Linux Pipes Are Slow

#65
post #58
post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

At some point, I had a similar issue (though not related to licensing), and it turned out it was faster to do a high-bitrate H.264-encode of the stream before sending it over the FFmpeg socket than sending the raw RGBA data, even over localhost… (There was some minimal quality loss, of course, but it was completely irrelevant in the big picture.)

> There was some minimal quality loss, of course, but it was completely irrelevant in the big picture

But then the solutions are not comparable anymore, are they? Would a lossless codec instead have improved speed?

Re: Linux Pipes Are Slow

#68
post #65
post #58

Earlier quoted context omitted.

At some point, I had a similar issue (though not related to licensing), and it turned out it was faster to do a high-bitrate H.264-encode of the stream before sending it over the FFmpeg socket than sending the raw RGBA data, even over localhost… (There was some minimal quality loss, of course, but it was completely irrelevant in the big picture.)

> There was some minimal quality loss, of course, but it was completely irrelevant in the big picture But then the solutions are not comparable anymore, are they? Would a lossless codec instead have improved speed?

H.264 has lossless mode.

Re: Linux Pipes Are Slow

#69
post #25

Earlier quoted context omitted.

I have a project that uses a proprietary SDK for decoding raw video. I output the decoded data as pure RGBA in a way FFMpeg can read through a pipe to re-encode the video to a standard codec. FFMpeg can't include the Non-Free SDK in their source, and it would be wildly impracticable to store the pure RGBA in a file. So pipes are the only way to do it, there are valid reasons to use high throughput pipes.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

ffmpeg's library is notorious for being a complete and utter mess

Re: Linux Pipes Are Slow

#70
post #42

How do you gather profiling information for kernel function calls from a user program?

I'll write an article on the flamegraphs specifically, but to get the data, just follow Julia's article! https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/

Could you clarify how are you testing the speed of the first example where you are not writing anything to stdout? Thanks.
Post reply on HN