Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

Post reply on HN
Print view View original

111–116 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#111

Post by vacuity »

jcrites wrote: post #30

Are there good data handling libraries that provide abstractions over pipes, sockets, files, and memory and implement optimizations like these? I'd be interested in knowing if there are such libraries in C, C++, Rust, or other systems languages. I wasn't familiar with some of the APIs mentioned in the article like splice() and vmsplice(), so I wondered if there are libraries that I might use when building ~low-level…

You might want to look at Cosh[1]. I'm puzzling over the paper right now, actually! It's a model for providing a message-passing abstraction that still allows for optimizations. I don't think it's really known outside of the research setting, and writing an efficient Cosh implementation will probably require some time.

In short, it provides three modes of transfer: move, share, and copy. For instance, a move transfer takes data that the sender has R/W permissions to and wholly "gives" it to the receiver. This may be done with page table VM remappings. It also has a strong or weak property that indicates whether the sender and receiver can be trusted to cooperate or must be strictly corralled with VM permission remappings.

To be honest, I don't know if it can be optimized well enough to match ultra-optimized pipes or whatever reliably. That might be a "sufficiently smart compiler" issue. Still, I think it's worth a shot.

[1] https://barrelfish.org/publications/trios14-baumann-cosh.pdf

Re: How fast are Linux pipes anyway? (2022)

#112

Post by dang »

crabbone wrote: post #88
nerdponx wrote: post #84

Earlier quoted context omitted.

> The problem was I wrote the parser in C. This is what enabled good performance. When my manager came back to work, she realized she declared that she doesn't know C and will never learn (even though she wasn't related directly to the project), and the project was thrown to the dogs. I sympathize with your manager here. If someone under me, ostensibly working on a Python app, wrote a component in C while I was away…

[flagged]

Personal attacks will get you banned here, so please don't post anything like this to HN.

If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.

Re: How fast are Linux pipes anyway? (2022)

#114

Post by Dylan16807 »

4death4 wrote: post #108
psd1 wrote: post #107

Earlier quoted context omitted.

Powershell pipelines are an engine construct rather than OS pipes or file descriptors. (If you include OS binaries in a PS pipeline, it will map the internal pipeline to OS pipes for that element of the pipeline, of course.) Every Powershell command has a begin, process, and end block. (If you don't write these explicitly, your code goes in an implicit end block.) When a pipeline is evaluated: 1. From left to right,…

Ok, so it sounds like Powershell would have the exact same issue as the Linux pipes. The issue has nothing to do with determinism with the pipeline construction and everything to do with the fact that part of the pipeline writes to stderr, which you could call stream 2.

The fact that echo green writes to stderr mainly just means that you can see the non-determinism happening, because if it wrote to stdout its output would be invisible.

The big part that's not deterministic is whether echo red succeeds or dies, along with which order the programs exit in. That would be nondeterministic even if you just ran "echo red | echo blue". But in that case you would always see "blue" so it would be hard to tell.

In powershell, it would be deterministic. It sounds like echo red would always succeed.

Re: How fast are Linux pipes anyway? (2022)

#115

Post by psd1 »

4death4 wrote: post #108
psd1 wrote: post #107

Earlier quoted context omitted.

Powershell pipelines are an engine construct rather than OS pipes or file descriptors. (If you include OS binaries in a PS pipeline, it will map the internal pipeline to OS pipes for that element of the pipeline, of course.) Every Powershell command has a begin, process, and end block. (If you don't write these explicitly, your code goes in an implicit end block.) When a pipeline is evaluated: 1. From left to right,…

Ok, so it sounds like Powershell would have the exact same issue as the Linux pipes. The issue has nothing to do with determinism with the pipeline construction and everything to do with the fact that part of the pipeline writes to stderr, which you could call stream 2.

You mean https://www.gibney.org/the_output_of_linux_pipes_can_be_inde... ?

Absolutely not, that would never happen in Powershell. I just explained how it works...?

Re: How fast are Linux pipes anyway? (2022)

#116

Post by TheFlyingFish »

boppo1 wrote: post #110
nesarkvechnep wrote: post #57

Earlier quoted context omitted.

In the end they'll just use a Lambda.

Is that beneficial or not? I'm still too much of a novice to know.

(assuming that we're talking about the AWS serverless functions) Like everything else, it's situational.

Upsides of lambdas are ease of deployment (no need to worry about servers, that's kind of the whole point of serverless), virtually infinite horizontal scaling, and a very generous free tier.

Downsides are relatively slow cold starts, difficulty of exposing to the outside world (eg via HTTP route), and lack of state management.

Personally I like using Lambda as glue between different parts of the AWS ecosystem, or to handle events, dispatch notifications etc.

However I would definitely not use Lambda for anything remotely resembling a stateful web app, for instance. The slow cold starts and inherent statelessness are going to make that difficult. Also API Gateway is a huge pain to work with, or was last time I looked at it.

Post reply on HN