Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

121–130 of 171 posts

Re: Linux Pipes Are Slow

#121

Post by qsantos »

ben-schaaf wrote: post #109
yas_hmaheshwari wrote: post #61

Earlier quoted context omitted.

This post has gone to the top of hacker news, so I think we should give him some slack Looks like an amazing article, and so much to learn on what happens under the hood

HN generates ~20k page views over the course of a day with a peak of 2k/h: https://harrisonbroadbent.com/blog/hacker-news-traffic-spike... . At ~1MB per page load - not sure how accurate this is, I don't think it fully loaded - this static blogpost requires 0.55MB/s to meet demand. An original raspberry pi B (10mpbs ethernet) on the average french mobile internet connection (8mbps) provides double that. I don't mean…

Totally agree, my server should definitely be able to handle the load. But this is a WordPress install, which is definitely doing too much work for what it is when just serving the pages. I plan to improve on this!

Re: Linux Pipes Are Slow

#122

Post by koverstreet »

immibis wrote: post #71
messe wrote: post #60

Earlier quoted context omitted.

> and for pipes, if both ends support using the ringbuffer they'll map the same ringbuffer Is there planned to be a standardized way to signal to the other end of the pipe that ring buffers are supported, so this could be handled transparently in libc? If not, I don't really see what advantage it gets you compared to shared memory + a futex for synchronization—for pipes that is.

Presumably the same interface still works if the other side is using read/write.

correct

Re: Linux Pipes Are Slow

#123

Post by epcoa »

duped wrote: post #36
jheriko wrote: post #2

just never use pipes. they are some weird archaism that need to die :P the only time ive used them is external constraints. they are just not useful.

I agree with this but with a much more nuanced take: avoid pipes if either reader or writer expects to do async i/o and you don't own both the reader and writer. In fact if you ever set O_NONBLOCK on a pipe you need to be damn sure both the reader and writer expect non-blocking i/o because you'll get heisenbugs under heavy i/o when either the reader/writer outpace each other and one expects blocking i/o. When's the l…

This isn't right. O_NONBLOCK doesn't mean the pipe doesn't stall, it just means you get an immediate errno and don't block on the syscall in the kernel waiting and this is specific to the file description, for which a pipe has 2 independent ones. Setting O_NONBLOCK on the writer does not affect the reader. If it did, this would break a ton of common use cases where pipelined programs are designed to not even know what is on the other side.

Not sure what printf has to do with, it isn't designed to be used with a non-block writer (but that only concerns one side). How will the reader being non-block change the semantics of the writer? It doesn't.

You can't set O_NONBLOCK on a pipe fd you expect to use with stdio, but that isn't unique to pipes. Whether the reader is O_NONBLOCK will not affect you if you're pushing the writer with printf/stdio.

(This is also a reason why I balk a bit when people refer to O_NONBLOCK as "async IO", it isn't the same and leads to this confusion)

Re: Linux Pipes Are Slow

#124

Post by ibern »

hi-v-rocknroll wrote: post #117
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

Citation needed. Pipes aren't used everywhere in production in hot paths. That just doesn't happen.

A lot of bioinformatics code relies very heavily on pipes.

Re: Linux Pipes Are Slow

#126

Post by Sohcahtoa82 »

2OEH8eoCRo0 wrote: post #103
paulannesley wrote: post #79

Earlier quoted context omitted.

Sometimes the best answer really is a faster Corolla! https://www.toyota.com/grcorolla/ (These machines have amazing engineering and performance, and their entire existence is a hack to work around rules making it unviable to bring the intended GR Yaris to the US market.. Maybe just enough eng/perf/hack/market relevance to HN folk to warrant my lighthearted reply. Also, the company president is still on the tools.

There's no replacement for displacement.

Apparently there is, because that car only has a 1.6L 3-cylinder engine and yet produces a whopping 300 horsepower.

Re: Linux Pipes Are Slow

#127

Post by 2OEH8eoCRo0 »

Sohcahtoa82 wrote: post #126
2OEH8eoCRo0 wrote: post #103

Earlier quoted context omitted.

There's no replacement for displacement.

Apparently there is, because that car only has a 1.6L 3-cylinder engine and yet produces a whopping 300 horsepower.

When? In the RPM sweet spot after waiting an eternity for the turbos to spool? There's always a catch.

Re: Linux Pipes Are Slow

#128

Post by bastawhiz »

0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I'm not sure that logic makes sense. Making a thing that's used ubiquitously a few percent faster it's absolutely a worthwhile investment of effort. Individual operations might but be very much faster but it's (in aggregate) a ton of electricity and time globally.

Re: Linux Pipes Are Slow

#129

Post by 0xbadcafebee »

bastawhiz wrote: post #128
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

I'm not sure that logic makes sense. Making a thing that's used ubiquitously a few percent faster it's absolutely a worthwhile investment of effort. Individual operations might but be very much faster but it's (in aggregate) a ton of electricity and time globally.

That's what's called premature optimization. Everywhere in our lives we do inefficient things. Despite the inefficiency we gain us something else: ease of use or access, simplicity, lower cost, more time, etc. The world and life as we know it is just a series of tradeoffs. Often optimization before it's necessary actually creates more drawbacks than benefits. When it's easy and has a huge benefit, or is necessary, then definitely optimize. It may be hard to accept this as a general principle, but in practice (mostly in hindsight) it becomes very apparent.

Donald Knuth thinks the same: https://en.wikipedia.org/wiki/Program_optimization#When_to_o...

Re: Linux Pipes Are Slow

#130

Post by saagarjha »

marcosdumay wrote: post #31
nitwit005 wrote: post #30

Earlier quoted context omitted.

If you don't think pipes offer much, don't use them. Saying "long function call" doesn't mean much since a function can take infinitely long.

A long distance function call, that invalidates everything on your cache.

…which is quite expensive.
Post reply on HN