Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

161–170 of 171 posts

Re: Linux Pipes Are Slow

#161

Post by guenthert »

arendtio wrote: post #84

I know pipes primarily from shell scripts. Are they being used in other contexts as extensively, too? Like C or Rust programs?

Most shells are C programs, so it's clearly possible to use pipes there (and consequently any language with a C FFI, including Rust) and it's done. It's cumbersome though. That's why there are so called glue languages, including shells. Compiler and similar tools might establish pipes and will do so directly, rather than via a glue language. Perhaps a grep through github could tell how 'extensive' this truly is.

For the data transfer rate it doesn't matter how (using which language) the pipe is established; C and Rust and the like will have a (small) edge up in the start-up time (latency) though.

Re: Linux Pipes Are Slow

#162

Post by michaelcampbell »

ndesaulniers wrote: post #137
fatcunt wrote: post #73

> I do not know why the JMP is not just a RET, however. This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret... > The NOP instructions at the beginning and at the end of the function allow ftrac…

There are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle

I believe there are a lot more of your 2nd point than you might think.

Re: Linux Pipes Are Slow

#164

Post by Sohcahtoa82 »

0xbadcafebee wrote: post #149
zackmorris wrote: post #140

Earlier quoted context omitted.

I didn't expect to be writing this comment on this article hah, but apparently there is such a thing called a surge tank for storing boost pressure to mostly eliminate turbo lag: https://www.highpowermedia.com/Archive/the-surge-tank https://forums.tdiclub.com/index.php?threads/air-tank-or-com... It's such an obvious idea that I'm kind of shocked it took them until 2003 to do it. Surely someone thought of this in like…

Afaik the general solutions to turbo lag are 1) a smaller turbo, 2) two turbos in stages, one spools earlier, the other later, 3) a transmission/differential/tune tailored to be in boost near-constantly during acceleration (something like a 10-speed cvt designed to keep you in high revs when accelerating in sport mode; not only keeps you in boost but is the ideal power band for these non-diesel boosted sports cars)

> 10-speed cvt

CVTs shouldn't even have a concept of "speeds". I absolutely hate how manufacturers will build cars with CVTs and then make them only go into discrete gear ratios. It completely destroys the entire reason for having a CVT.

I understand that they do it because people don't like how CVTs sound/feel, but maybe they should all have 3 modes:

1. Eco - optimizes gear ratio for maximum effeciency

2. Performance - optimizes for maximum power

3. Sport - pretends to be a normal transmission for a better "feel".

Re: Linux Pipes Are Slow

#165

Post by fragmede »

imtringued wrote: post #160
samastur wrote: post #142

Earlier quoted context omitted.

No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time. I admit my opinion is not based on first hand knowledge, but I have for years worked on projects trying to address poverty at different parts of this planet and can't think of a single one where this would be even remotely true.

>No, they really aren't. Absolutely nobody's life is measurably improved because of 1 cent one time. Assuming a wage of $35/hour, each second is worth 1 cent. To save 1 cent you only need to reduce the time spent waiting for computers by a second across the entire lifetime of that person. Now here is the beauty of this. There isn't just a single guy out there doing this. There are hundreds of thousands of people, pos…

The beauty of math is that you can throw numbers around and multiply and divide them and do silly things with them.

The average human life expectancy is 77.5 years, or 2.4457e+9 seconds. If you divide that by, say, 1 billion daily active users of Google, you get 2.445. So if you work at Google, and optimize a slow process, and save every user 1 second, once, you've saved 2 lives. If you're a Microsoft and make boot up take 1 second less across their billion or so devices, same thing.

Re: Linux Pipes Are Slow

#166

Post by koolala »

ehvatum wrote: post #163
koolala wrote: post #157

Earlier quoted context omitted.

How can I get hired to do bioinformatics with pipes all day? Sounds like a dream. I have VR and control systems experience.

A dream that pays $35k/yr

I'm ok with that if it means working with Pipes all day. Just needs affordable housing nearby... or my car...

Re: Linux Pipes Are Slow

#167

Post by nitwit005 »

brigade wrote: post #33
nitwit005 wrote: post #8

Just about every form of IPC is "slow". You have decided to pay a performance cost for safety.

Pipes don’t exist for safety, they exist as an optimization to pass data between existing programs.

The safety is memory protection. If you don't care about memory protection, you can reduce IPC to passing pointers around.

Re: Linux Pipes Are Slow

#168

Post by sk5t »

ndesaulniers wrote: post #137
fatcunt wrote: post #73

> I do not know why the JMP is not just a RET, however. This is caused by the CONFIG_RETHUNK option. In the disassembly from objdump you are seeing the result of RET being replaced with JMP __x86_return_thunk. https://github.com/torvalds/linux/blob/v6.1/arch/x86/include... https://github.com/torvalds/linux/blob/v6.1/arch/x86/lib/ret... > The NOP instructions at the beginning and at the end of the function allow ftrac…

There are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle

Have you accounted for the population of Australian kernel developers?

Re: Linux Pipes Are Slow

#169

Post by ndesaulniers »

sk5t wrote: post #168
ndesaulniers wrote: post #137

Earlier quoted context omitted.

There are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle

Have you accounted for the population of Australian kernel developers?

Good point; in that case I may have undercounted! :P

That said, most Aussie kernel devs I know are IBM PPC Ozlabs folks that are actually really nice!

Re: Linux Pipes Are Slow

#170

Post by opello »

michaelcampbell wrote: post #162
ndesaulniers wrote: post #137

Earlier quoted context omitted.

There are perhaps only a handful of kernel developers that: 1. would know the above 2. would choose such an obnoxious throwaway handle

I believe there are a lot more of your 2nd point than you might think.

And certainly those interested in the performance implications of the retpoline mitigation for the various speculative execution information leaks.
Post reply on HN