Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

Post reply on HN
Print view View original

61–70 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#61

Post by nikanj »

winternewt wrote: post #51
Too wrote: post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Your coworkers would prefer you splitting the thing into two microservices communicating over a REST api, aka the 200x slower version.

Re: How fast are Linux pipes anyway? (2022)

#62

Post by hgraves1991 »

nikanj wrote: post #61
winternewt wrote: post #51

Earlier quoted context omitted.

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Your coworkers would prefer you splitting the thing into two microservices communicating over a REST api, aka the 200x slower version.

We hate to see it

Re: How fast are Linux pipes anyway? (2022)

#63

Post by 127 »

loondri wrote: post #60

This article talks about making Linux pipes faster, but other methods like shared memory or message queues might still be quicker. For example, in systems that need to move a lot of data quickly, the extra steps with pipes could slow things down. Also, when many threads are sharing data, pipes might cause more problems than other methods. So, the improvements in the article might not help much in real-world situation…

Also the benefit of using a message queue library is that you don't have to worry about multi-platform incompatibilities as much.

Re: How fast are Linux pipes anyway? (2022)

#64

Post by 0d0a »

loondri wrote: post #60

This article talks about making Linux pipes faster, but other methods like shared memory or message queues might still be quicker. For example, in systems that need to move a lot of data quickly, the extra steps with pipes could slow things down. Also, when many threads are sharing data, pipes might cause more problems than other methods. So, the improvements in the article might not help much in real-world situation…

Can you give some examples? When batching data, you benefit from picking something like io_uring. But for two-way communication, you still need to notify either side when data is ready (maybe you don't want to consume cpu just polling), and it isn't clear to me how those options handle that synchronization faster than pipes.

Re: How fast are Linux pipes anyway? (2022)

#65

Post by NavinF »

duped wrote: post #37
jcrites wrote: post #30

Are there good data handling libraries that provide abstractions over pipes, sockets, files, and memory and implement optimizations like these? I'd be interested in knowing if there are such libraries in C, C++, Rust, or other systems languages. I wasn't familiar with some of the APIs mentioned in the article like splice() and vmsplice(), so I wondered if there are libraries that I might use when building ~low-level…

This may go against the grain but this isn't really worth abstracting over since it's not portable. You'll probably want to implement it by hand everywhere you need it. Higher level code only uses them rarely because they're pretty special purpose and they have to be specialized for Linux. If you're shuffling data around without looking at it only on Linux, splice is useful. There's not that many applications that ha…

The main reason why people write abstractions over stuff like this is to make it portable. I'm sure there's something similar to vmsplice on every relevant OS. The library can also fallback to write_read if you're targeting some ancient platform

Re: How fast are Linux pipes anyway? (2022)

#66

Post by xxs »

winternewt wrote: post #51
Too wrote: post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

it's time to change your workplace, not everyone is meant to be petty/incompetent.

Re: How fast are Linux pipes anyway? (2022)

#67

Post by tucnak »

dilyevsky wrote: post #32
gpderetta wrote: post #17

Earlier quoted context omitted.

Pipes are zero copy only if you use splice or vmsplice. These linux specific syscalls are hard to use (particularly wmsplice) and the vast majority of programs and shell filters (with the notable exception of pv) don't use them and pay for the cost of copying in and out of kernel memory.

If you’re using Go it will automatically splice your reader/writer when using io.Copy, etc

re: https://go.dev/src/net/splice_linux.go

very interesting, I didn't know `io` was doing that on linux!

Re: How fast are Linux pipes anyway? (2022)

#68

Post by jstimpfle »

rixed wrote: post #50
jstimpfle wrote: post #5

Earlier quoted context omitted.

Not surprising, the pipe you've created doesn't transport any of the data you've echoed. (echo red; echo green 1>&2) | echo blue This creates two subshells separated by the pipe | symbol. A subshell is a child process of the current shell, and as such it inherits important properties of the current shell, notably including the open file descriptor table. Since they are child processes, both subshells run concurrently…

I don't think your message (or others) does justice to the original blogpost. Yes the pipe runs two subcommands in parallel but that is not why the blogpost is interesting (or its author surprised). It's because 'echo red' is supposed to block , thus introducing synchronization between the two branches of the pipe, yet it doesn't! And I must confess, when reading the command my first though was: "Ok so that first ech…

Yes, I noticed that only after finishing the work on my comment (which, strangely enough, is my most-upvoted comment ever). I had been under the impression that the command is a construction from a beginner trying to make sense of the shell, so I skipped over the blogpost too quickly.

But indeed the author wasn't aware that readers and witers of the pipe aren't fully synchronized because the buffer in between allows for some concurrency. My writeup wasn't very explicit about that (at least not that writing to the pipe can block when the pipe is full) but I think it's technically accurate and hope it can clear up some confusion -- a lot of readers probably do not understand well how the shell works.

Re: How fast are Linux pipes anyway? (2022)

#69

Post by _trackno5 »

0d0a wrote: post #64
loondri wrote: post #60

This article talks about making Linux pipes faster, but other methods like shared memory or message queues might still be quicker. For example, in systems that need to move a lot of data quickly, the extra steps with pipes could slow things down. Also, when many threads are sharing data, pipes might cause more problems than other methods. So, the improvements in the article might not help much in real-world situation…

Can you give some examples? When batching data, you benefit from picking something like io_uring. But for two-way communication, you still need to notify either side when data is ready (maybe you don't want to consume cpu just polling), and it isn't clear to me how those options handle that synchronization faster than pipes.

The main thing io_uring gives you is avoiding multiple syscalls.

With a pipe you can’t really avoid that. With a shared memory queue/ring buffer you can write to the memory without any syscalls.

But you need to build synchronisation yourself (e.g., using semaphores for example). You don’t necessarily need to poll.

Post reply on HN