Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

151–160 of 171 posts

Re: Linux Pipes Are Slow

#151

Post by bastawhiz »

0xbadcafebee wrote: post #148
bastawhiz wrote: post #132

Earlier quoted context omitted.

It's definitionally not premature optimization. Pipes exist (and have existed for decades). This is just "optimization". "Premature" means it's too soon to optimize. When is it no longer too soon? In another few decades? When Linux takes another half of Windows usage? It would be premature if they were working on optimizations before there were any users or a working implementation. But it's not: they're a fundamenta…

Most people will never need to optimize pipes. If they don't need to do it, it's premature. If they need to do it, it's not premature

If you're saying that people that know they don't need to optimize pipes shouldn't optimize pipes, then yeah I mean that's kind of just common sense.

Re: Linux Pipes Are Slow

#152

Post by whiterknight »

CyberDildonics wrote: post #131
whiterknight wrote: post #98

Earlier quoted context omitted.

And you go from having a well defined modular interface that’s flexible at runtime to a binary dependency.

You have the dependency either way, but if you use the library you can have one big executable with no external dependencies and it can actually be fast. If there wasn't a problem to solve they wouldn't have said anything. If you want something different you have to do something different.

The context of this discussion is that it would be better if pipes were faster. Then you would have more options.

Re: Linux Pipes Are Slow

#153

Post by seba_dos1 »

quietbritishjim wrote: post #72
CyberDildonics wrote: post #32

Earlier quoted context omitted.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

The parent comment mentioned license incompatibility, which I guess would still apply if they used ffmpeg as a library.

If the license is incompatible, it would still be incompatible regardless of whether you use library API calls or pipes.

Re: Linux Pipes Are Slow

#154

Post by CyberDildonics »

whiterknight wrote: post #152
CyberDildonics wrote: post #131

Earlier quoted context omitted.

You have the dependency either way, but if you use the library you can have one big executable with no external dependencies and it can actually be fast. If there wasn't a problem to solve they wouldn't have said anything. If you want something different you have to do something different.

The context of this discussion is that it would be better if pipes were faster. Then you would have more options.

I replied to them saying "So pipes are the only way to do it".

Re: Linux Pipes Are Slow

#156

Post by sebstefan »

samastur wrote: post #143
sebstefan wrote: post #81

Earlier quoted context omitted.

So? It doesn't need to be visible to be worth optimizing?

If you are making an economic (financial) argument for change like the original comment did, then yes, it should be visible positive effect. Obviously not if you are doing for your own fun or just improving the state of art.

Eventually everything ends up pumping the same resources from the same earth. The billion devices pool their 1 cents saved into the same pool of fossil fuels and the same power plants.

You don't need the effect to be observable on an individual level

It's something that is worth an engineer's time

Re: Linux Pipes Are Slow

#157

Post by koolala »

ibern wrote: post #124
hi-v-rocknroll wrote: post #117

Earlier quoted context omitted.

Citation needed. Pipes aren't used everywhere in production in hot paths. That just doesn't happen.

A lot of bioinformatics code relies very heavily on pipes.

How can I get hired to do bioinformatics with pipes all day? Sounds like a dream. I have VR and control systems experience.

Re: Linux Pipes Are Slow

#158

Post by WhyNotHugo »

koverstreet wrote: post #12

One of my sideprojects is intended to address this: https://lwn.net/Articles/976836/ The idea is a syscall for getting a ringbuffer for any supported file descriptor, including pipes - and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer: zero copy IO, potentially without calling into the kernel at all. Would love to find collaborators for this one :)

I wonder if existing ring buffer interfaces will consider using this or if we'll see an xkcd927 situation. Regardless, this seems like an interesting endeavour.

Re: Linux Pipes Are Slow

#159

Post by imtringued »

samastur wrote: post #76
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

An investment or economic action is economically viable when societal benefits exceed the initial capital outlay. However, in capitalism people care more about the personal return on the initial capital outlay, than the actual societal benefits of their investment.

If society was a giant hivemind, then economic viability would take precedence over personal profit. Meanwhile if society is a bunch of isolated individuals, economic viability would take the backseat. So this tells us more about the limits of human psychology than it tells us about economics.

Re: Linux Pipes Are Slow

#160

Post by imtringued »

samastur wrote: post #142
h0p3 wrote: post #87

Earlier quoted context omitted.

There are people whose lives are improved by having an extra cent to spend. Seriously. It is measurable, observable, and real. It might not have a serious impact on the vast majority of people, but there are people who have very, very little money or have found themselves on a tipping point that small; pinching pennies alters their utility outcomes.

No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time. I admit my opinion is not based on first hand knowledge, but I have for years worked on projects trying to address poverty at different parts of this planet and can't think of a single one where this would be even remotely true.

>No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time.

Assuming a wage of $35/hour, each second is worth 1 cent. To save 1 cent you only need to reduce the time spent waiting for computers by a second across the entire lifetime of that person.

Now here is the beauty of this. There isn't just a single guy out there doing this. There are hundreds of thousands of people, possibly millions, doing it.

Post reply on HN