Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

131–140 of 171 posts

Re: Linux Pipes Are Slow

#131

Post by CyberDildonics »

whiterknight wrote: post #98
CyberDildonics wrote: post #32

Earlier quoted context omitted.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

And you go from having a well defined modular interface that’s flexible at runtime to a binary dependency.

You have the dependency either way, but if you use the library you can have one big executable with no external dependencies and it can actually be fast.

If there wasn't a problem to solve they wouldn't have said anything. If you want something different you have to do something different.

Re: Linux Pipes Are Slow

#132

Post by bastawhiz »

0xbadcafebee wrote: post #129
bastawhiz wrote: post #128

Earlier quoted context omitted.

I'm not sure that logic makes sense. Making a thing that's used ubiquitously a few percent faster it's absolutely a worthwhile investment of effort. Individual operations might but be very much faster but it's (in aggregate) a ton of electricity and time globally.

That's what's called premature optimization. Everywhere in our lives we do inefficient things. Despite the inefficiency we gain us something else: ease of use or access, simplicity, lower cost, more time, etc. The world and life as we know it is just a series of tradeoffs. Often optimization before it's necessary actually creates more drawbacks than benefits. When it's easy and has a huge benefit, or is necessary, th…

It's definitionally not premature optimization. Pipes exist (and have existed for decades). This is just "optimization". "Premature" means it's too soon to optimize. When is it no longer too soon? In another few decades? When Linux takes another half of Windows usage? It would be premature if they were working on optimizations before there were any users or a working implementation. But it's not: they're a fundamental primitive of the OS used by tens of millions of applications.

The tradeoffs you're discussing are considerations. Is it worth making a ubiquitous thing faster at the expense of some complexity? At some point that answer is "yes", but that is absolutely not "When it's easy and has a huge benefit". The most important optimizations you personally benefit from were not easy OR had a huge benefit. They were hard won and generally small, but they compound on other optimizations.

I'll also note that the Knuth quote you reference says exactly this:

> Yet we should not pass up our opportunities in that critical 3%

Re: Linux Pipes Are Slow

#134

Post by lillecarl »

wang_li wrote: post #113
lillecarl wrote: post #107

Earlier quoted context omitted.

Eventually everyone ends up at a power plant, there's an insane amount of people living in the European grid. If an optimization ends up saving a couple tonnes of CO2 per year it is hard to not call it a good thing. https://en.m.wikipedia.org/wiki/Synchronous_grid_of_Continen...

A couple tons spread across 400 million people with a per capita emission of 5 tons per year is in the noise. If we're at the point of trying to hyper optimize there are far more meaningful targets than pipe throughput.

Absolutely, if your focus is saving emissions don't optimize pipes. But if you optimize an interface people use it's a good thing either way right

Re: Linux Pipes Are Slow

#135

Post by tacone »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Wait, it depends on what you're doing. Pipes also create a subshell so they are a big nono when used inside a loop.

Suppose you're cycling on the lines of stdout and need to use sed, cut and so on, using pipes will slow down things considerably (and sed, cut startup time will make things worse).

Using bash/zsh string interpolation would be much faster.

Re: Linux Pipes Are Slow

#136

Post by yencabulator »

FUSE can be a bit trickier than a single queue of data chunks. Reads from /dev/fuse actually pick the right message to read based on priorities, and there's cases where the message queue is meddled with to e.g. cancel requests before they're even sent to userspace. If you naively switch it to eagerly putting messages into a userspace-visible ringbuffer, you might significantly change behavior in cases like interrupting slow operations. Imagine having to fulfill a ringbuf worth of requests to a misbehaving backend taking 5sec/op, just to see the cancellations at the very end.

Re: Linux Pipes Are Slow

#137

Post by ndesaulniers »

fatcunt wrote: post #73

> I do not know why the JMP is not just a RET, however. This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret... > The NOP instructions at the beginning and at the end of the function allow ftrac…

There are perhaps only a handful of kernel developers that:

1. would know the above

2. would choose such an obnoxious throwaway handle

Re: Linux Pipes Are Slow

#138

Post by throwway120385 »

hi-v-rocknroll wrote: post #117
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

Citation needed. Pipes aren't used everywhere in production in hot paths. That just doesn't happen.

What production? You need to check your assumptions about what people do with general purpose computers and why. Just because it doesn't happen in your specific field of computing doesn't mean it never happens anywhere or that it just isn't important.

Re: Linux Pipes Are Slow

#139

Post by ismaildonmez »

qsantos wrote: post #89
ismaildonmez wrote: post #70

Earlier quoted context omitted.

Could you clarify how are you testing the speed of the first example where you are not writing anything to stdout? Thanks.

For the first Rust program, where I just write to memory, I just use the time utility when running the program from zsh. Then, I divide the number of bytes written by the number of seconds elapsed. That's why it's not an infinite loop ;)

Thanks!

Re: Linux Pipes Are Slow

#140

Post by zackmorris »

2OEH8eoCRo0 wrote: post #127
Sohcahtoa82 wrote: post #126

Earlier quoted context omitted.

Apparently there is, because that car only has a 1.6L 3-cylinder engine and yet produces a whopping 300 horsepower.

When? In the RPM sweet spot after waiting an eternity for the turbos to spool? There's always a catch.

I didn't expect to be writing this comment on this article hah, but apparently there is such a thing called a surge tank for storing boost pressure to mostly eliminate turbo lag:

https://www.highpowermedia.com/Archive/the-surge-tank

https://forums.tdiclub.com/index.php?threads/air-tank-or-com...

It's such an obvious idea that I'm kind of shocked it took them until 2003 to do it. Surely someone thought of this in like the 60s.

I would probably do it differently with a separate supercharger to intermittently maintain another 1-2+ bar of boost to make the tank less than half as large, but that would add complexity, and what do I know.

Post reply on HN