Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Pipes Are Slow

qsantos.fr

Post reply on HN
Print view View original

101–110 of 171 posts

Re: Linux Pipes Are Slow

#101

Post by InDubioProRubio »

h0p3 wrote: post #87
samastur wrote: post #76

Earlier quoted context omitted.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

There are people whose lives are improved by having an extra cent to spend. Seriously. It is measurable, observable, and real. It might not have a serious impact on the vast majority of people, but there are people who have very, very little money or have found themselves on a tipping point that small; pinching pennies alters their utility outcomes.

https://xkcd.com/951/

Also, if you micro-optimize and that becomes your whole focus and ability to focus, your business is unable to innovate aka traverse the economic landscape and find new rich gradients and sources of "economic food", making you a dinosaur in a pit, doomed to eternally cannibalize on what other creatures descend into the pit and highly dependent on the pit not closing up for good.

Re: Linux Pipes Are Slow

#102

Post by CyberDildonics »

Almondsetat wrote: post #69
CyberDildonics wrote: post #32

Earlier quoted context omitted.

So pipes are the only way to do it Lets not get carried away. You can use ffmpeg as a library and encode buffers in a few dozen lines of C++.

ffmpeg's library is notorious for being a complete and utter mess

It worked extremely well when I did something almost exactly like this. I gave it buffers of pixels in memory and it spit out compressed video.

Re: Linux Pipes Are Slow

#103

Post by 2OEH8eoCRo0 »

paulannesley wrote: post #79
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Sometimes the best answer really is a faster Corolla! https://www.toyota.com/grcorolla/ (These machines have amazing engineering and performance, and their entire existence is a hack to work around rules making it unviable to bring the intended GR Yaris to the US market.. Maybe just enough eng/perf/hack/market relevance to HN folk to warrant my lighthearted reply. Also, the company president is still on the tools.

There's no replacement for displacement.

Re: Linux Pipes Are Slow

#104

Post by IWeldMelons »

throw12390 wrote: post #74
tzury wrote: post #67

Earlier quoted context omitted.

pv https://linux.die.net/man/1/pv it is in the pipe command `... | pv > /dev/null`

`pv --discard` is faster by 8% (on my system). % pv /dev/null 54.0GiB/s % pv

Which is suspiciously close to the speed of DDR4.

Re: Linux Pipes Are Slow

#105

Post by wang_li »

whiterknight wrote: post #99
samastur wrote: post #76

Earlier quoted context omitted.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

You’re describing the outcome of one individual person. Money is just a tool for allocating resources. Saving 1 million of resources is a good thing.

It's a meaningless thing if it's 1 million resources divided into 1 million actors who have no ability to leverage a short term gain of 1 resource. It's short term because the number of computers that are 100% busy 100% of the time is zero. A pipe throughput improvement means nothing if the computer isn't waiting on pipes a lot.

Re: Linux Pipes Are Slow

#106

Post by Narishma »

donaldihunter wrote: post #100

Something I didn't see mentioned in the article about AVX512, aside from the xsave/xrstor overhead, is that AVX512 is power hungry and causes CPU frequency scaling. See [1], [2] for details and as an example of how nuanced it can get. [1] https://www.intel.com/content/dam/www/central-libraries/us/e... [2] https://www.intel.com/content/www/us/en/developer/articles/t...

That is only the case in specific Intel CPU models.

Re: Linux Pipes Are Slow

#107

Post by lillecarl »

wang_li wrote: post #105
whiterknight wrote: post #99

Earlier quoted context omitted.

You’re describing the outcome of one individual person. Money is just a tool for allocating resources. Saving 1 million of resources is a good thing.

It's a meaningless thing if it's 1 million resources divided into 1 million actors who have no ability to leverage a short term gain of 1 resource. It's short term because the number of computers that are 100% busy 100% of the time is zero. A pipe throughput improvement means nothing if the computer isn't waiting on pipes a lot.

Eventually everyone ends up at a power plant, there's an insane amount of people living in the European grid. If an optimization ends up saving a couple tonnes of CO2 per year it is hard to not call it a good thing.

https://en.m.wikipedia.org/wiki/Synchronous_grid_of_Continen...

Re: Linux Pipes Are Slow

#108

Post by azulster »

samastur wrote: post #76
Someone wrote: post #40

Earlier quoted context omitted.

This isn’t code in some project that will run only a few billion times in its lifetime; it is used frequently on millions, if not billions, of computers. Because of that, it is economical to spend lots of time optimizing it, even if it only makes the code marginally more efficient.

That's not how economics works. If 100 million people each save 1 cent because of your work, you saved 1 million in total, but in practice nobody is observably better off.

not if it costs 200 million in man-hours to optimize

Re: Linux Pipes Are Slow

#109

Post by ben-schaaf »

yas_hmaheshwari wrote: post #61
stabbles wrote: post #59

A bold claim for a blog that takes about 20 seconds to load.

This post has gone to the top of hacker news, so I think we should give him some slack Looks like an amazing article, and so much to learn on what happens under the hood

HN generates ~20k page views over the course of a day with a peak of 2k/h: https://harrisonbroadbent.com/blog/hacker-news-traffic-spike.... At ~1MB per page load - not sure how accurate this is, I don't think it fully loaded - this static blogpost requires 0.55MB/s to meet demand. An original raspberry pi B (10mpbs ethernet) on the average french mobile internet connection (8mbps) provides double that.

I don't mean this as a slight to anyone, I just want to point out the HN "hug of death" can be trivially handled by a single cheap VPS without even breaking a sweat.

Re: Linux Pipes Are Slow

#110

Post by azulster »

jiehong wrote: post #77
0xbadcafebee wrote: post #13

Calling Linux pipes "slow" is like calling a Toyota Corolla "slow". It's fast enough for all but the most extreme use cases. Are you racing cars? In a sport where speed is more important than technique? Then get a faster car. Otherwise stick to the Corolla.

Replace “Linux pipes” by “Electron apps”, and people would not agree. Also, why leave performance on the table by default? Just because “it should be enough for most people I can think of”? Add Tesla motors to a Toyota Corolla and now you’ve got a sportier car by default.

electron apps are an optimization all by itself.

it's not optimizing footprint or speed of application. it's optimizing the resources and speed of development and deployment

Post reply on HN