Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

141–150 of 171 posts

Re: Linux Pipes Are Slow

#141

Earlier quoted context omitted.

A long distance function call, that invalidates everything on your cache.

…which is quite expensive.

Yes, it is. But it's much cheaper than interacting by pipe.

Linux is optimizing sockets with a similar goal. And it's quite far on that direction. But there's still some margin to gain.

Re: Linux Pipes Are Slow

#142
post #87

Earlier quoted context omitted.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

There are people whose lives are improved by having an extra cent to spend. Seriously. It is measurable, observable, and real. It might not have a serious impact on the vast majority of people, but there are people who have very, very little money or have found themselves on a tipping point that small; pinching pennies alters their utility outcomes.

No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time.

I admit my opinion is not based on first hand knowledge, but I have for years worked on projects trying to address poverty at different parts of this planet and can't think of a single one where this would be even remotely true.

Re: Linux Pipes Are Slow

#143

Earlier quoted context omitted.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

So? It doesn't need to be visible to be worth optimizing?

If you are making an economic (financial) argument for change like the original comment did, then yes, it should be visible positive effect.

Obviously not if you are doing for your own fun or just improving the state of art.

Re: Linux Pipes Are Slow

#144
post #87

Earlier quoted context omitted.

There are people whose lives are improved by having an extra cent to spend. Seriously. It is measurable, observable, and real. It might not have a serious impact on the vast majority of people, but there are people who have very, very little money or have found themselves on a tipping point that small; pinching pennies alters their utility outcomes.

No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time. I admit my opinion is not based on first hand knowledge, but I have for years worked on projects trying to address poverty at different parts of this planet and can't think of a single one where this would be even remotely true.

> Absolutely nobody's life is measurably improved because of 1 cent one time...I admit my opinion is not based on first hand knowledge...

My opinion, however, is based on first-hand knowledge. I've been the kid saving those pennies, and I've worked with those kids. I understand that in the vast majority of cases, an extra penny does nothing more. That isn't what your original comment above claimed, nor is it what you've claimed here. My counterexample is enough to demonstrate the falsehood. Arguing that there are better ways to distribute these pennies is another matter, and I take that seriously as well.

Re: Linux Pipes Are Slow

#145
post #56

Earlier quoted context omitted.

What about domain sockets? It's clumsier, to be sure, but if performance is your goal, the socket should be faster.

Why should sockets be faster?

Sockets remap pages without moving any data while pipes have to copy the data between fds.

Re: Linux Pipes Are Slow

#146
post #123
post #36

Earlier quoted context omitted.

I agree with this but with a much more nuanced take: avoid pipes if either reader or writer expects to do async i/o and you don't own both the reader and writer. In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the l…

This isn't right. O_NONBLOCK doesn't mean the pipe doesn't stall, it just means you get an immediate errno and don't block on the syscall in the kernel waiting and this is specific to the file description, for which a pipe has 2 independent ones. Setting O_NONBLOCK on the writer does not affect the reader. If it did, this would break a ton of common use cases where pipelined programs are designed to not even know wha…

It's an unfortunately common bug for a process to set non blocking on a file shared with another process.

Re: Linux Pipes Are Slow

#147

This is a side note to the main point being made, but on modern CPUs, "rep movsb" is just as fast as the fastest vectorized version, because the CPU knows to accelerate it. The name of the kernel function "copy_user_enhanced_fast_string" hints at this: the CPU features are ERMS ("Enhanced Repeat Move String", which makes "rep movsb" faster for anything above a certain length threshold) and FSRM ("Fast Short Repeat Mo…

If that’s the case, when can I expect C compilers to inline variable-length memcpy() the way they will inline fixed-length memcpy today?

Re: Linux Pipes Are Slow

#148

Earlier quoted context omitted.

That's what's called premature optimization. Everywhere in our lives we do inefficient things. Despite the inefficiency we gain us something else: ease of use or access, simplicity, lower cost, more time, etc. The world and life as we know it is just a series of tradeoffs. Often optimization before it's necessary actually creates more drawbacks than benefits. When it's easy and has a huge benefit, or is necessary, th…

It's definitionally not premature optimization. Pipes exist (and have existed for decades). This is just "optimization". "Premature" means it's too soon to optimize. When is it no longer too soon? In another few decades? When Linux takes another half of Windows usage? It would be premature if they were working on optimizations before there were any users or a working implementation. But it's not: they're a fundamenta…

Most people will never need to optimize pipes. If they don't need to do it, it's premature. If they need to do it, it's not premature

Re: Linux Pipes Are Slow

#149

Earlier quoted context omitted.

When? In the RPM sweet spot after waiting an eternity for the turbos to spool? There's always a catch.

I didn't expect to be writing this comment on this article hah, but apparently there is such a thing called a surge tank for storing boost pressure to mostly eliminate turbo lag: https://www.highpowermedia.com/Archive/the-surge-tank https://forums.tdiclub.com/index.php?threads/air-tank-or-com... It's such an obvious idea that I'm kind of shocked it took them until 2003 to do it. Surely someone thought of this in like…

Afaik the general solutions to turbo lag are 1) a smaller turbo, 2) two turbos in stages, one spools earlier, the other later, 3) a transmission/differential/tune tailored to be in boost near-constantly during acceleration (something like a 10-speed cvt designed to keep you in high revs when accelerating in sport mode; not only keeps you in boost but is the ideal power band for these non-diesel boosted sports cars)

Re: Linux Pipes Are Slow

#150
post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

It's because of you that my phone is so slow.
Post reply on HN