Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

91–100 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#91

Earlier quoted context omitted.

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

> And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player. Hear hear! Why is it like this?

Because spending tens of developer-hours to save tens of compute-hours usually isn't worth it. Justify to me that it's worth the time investment, maintenance burden, and risk of failure, then I'll let you work on it.

Re: How fast are Linux pipes anyway? (2022)

#92
post #66

Earlier quoted context omitted.

it's time to change your workplace, not everyone is meant to be petty/incompetent.

My experience is that even the most committed and smart people degrade as teams into patterns which produce seemingly unnecessary complexity because that's where our industry trends and tools push us. People create boxes to isolate perceived complexity, that in turn create new complexity. I share the other commenter's frustrations. I want out of the tarpit.

I have the exactly opposite experience - I call bullcrap when I see it, bluntly and directly. 'Industry standard' nonsense is still nonsense.

The sheer truth is that many developers are just CV driven, or they miss their childhood playing with toys. Overall the software developers/engineers (we) are an extremely spoiled breed, often entirely detached from reality - and it shows.

Re: How fast are Linux pipes anyway? (2022)

#93

Earlier quoted context omitted.

> The problem was I wrote the parser in C. This is what enabled good performance. When my manager came back to work, she realized she declared that she doesn't know C and will never learn (even though she wasn't related directly to the project), and the project was thrown to the dogs. I sympathize with your manager here. If someone under me, ostensibly working on a Python app, wrote a component in C while I was away…

[flagged]

[flagged]

Re: How fast are Linux pipes anyway? (2022)

#95

Earlier quoted context omitted.

This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.

Pipes are FIFO data buffers implemented in the kernel. For communication between threads of the same process, you can replace any pipe object by a userspace queue implementation protected by e.g. mutex + condition variable. It is functionally equivalent and has potential to be faster. And if you wrap all accesses in lock/unlock pairs (without locking any other objects in between) there is no danger of introducing any…

There are domain sockets if you need something more such as passing file descriptors. Both pipes and sockets (including TCP, with obvious limitations) can be done with zero copy given the right set of flags, thought things get harder if you have a complicated runtime (ie. garbage collection) involved. There's always explicitly mapped shared pages

Re: How fast are Linux pipes anyway? (2022)

#96
post #61

Earlier quoted context omitted.

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Your coworkers would prefer you splitting the thing into two microservices communicating over a REST api, aka the 200x slower version.

A REST api without a hint of hypertext.

Re: How fast are Linux pipes anyway? (2022)

#97

Earlier quoted context omitted.

I have a very long list of things that were good, worked well, and ended up rejected because the team didn't want to put in effort to learn how they work. My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but bei…

A couple of things stand out from the stories you mentioned. I should say first of all that I do sympathize and when I was starting out I definitely did do similar stuff to what you mentioned. 1) Whenever a new framework is introduced this usually requires broad consent, clear scope, and agreement on committing to this new direction, doubly so when it's introducing a new language. Do you embark on working on these pr…

> Whenever a new framework is introduced this usually requires broad consent, clear scope, and agreement on committing to this new direction, doubly so when it's introducing a new language. Do you embark on working on these projects without discussing what you'll be doing?

Like I mentioned above. If I want quality, then I alone make this decision. If anyone wants to join, they join on my terms. I will make my terms uncompromising and uncomfortable for people who want consensus deliberately because I don't want to work with people who want consensus -- they suck at programming.

For my day job, I work with people who want consensus, an easy way to slack, pretend to work, hit the lowest bar available to pick the paycheck and go home, watch TV, play guitar, whatever. They don't produce anything that resembles quality work. Never will. That's not their goal, nor do they feel bad for not accomplishing that.

> Who's going to maintain that? Why wasn't this discussed?

Whoever knows how to program will.

It wasn't discussed because I wouldn't care about an opinion coming from someone who doesn't know how to program. Same way how I don't ask the neighbor's cat when I make programming decisions.

> Have you even considered code reviews? ... if no one else knows the language and _knows it well_?

Yes. I considered. This is their problem, not mine. If you want to be a programmer, it's your job to know your tools well. If you want a fat paycheck and be a useless blight on the bloat of the corporate world... well, make up the rules that are even more convenient to you...

> in love with tech

This has nothing to do with being "in love with tech". This is about the quality of output of people who are employed as programmers but aren't. One can be "in love" and suck, while other can "hate" and be very good at what they do. The reason why the situation the way it is that that human nature which pushes everyone towards a place where they can be lazy and ignorant meets no resistance.

While in many other professions there's a market for high-quality products, like very expensive and very high-quality watches, cars, photo equipment, clothes, food... in programming there's no market for anything that's trying for quality rather than time to market, price or reach. There's no niche, when it comes to programming market that would pay tenfold or hundred times more for a higher-quality product. That's why in industrial setting nobody is trying to make high-quality products, even though some, naively, come with this idea into the trade, they are quickly shown the reality where nobody cares.

Re: How fast are Linux pipes anyway? (2022)

#98
post #40

Earlier quoted context omitted.

This is why threads aren't nearly as important as many programmers seem to think. Chances are, whatever application you're building can be done in a cleaner way using pipes + processes or green/user-space threads depending on the workload in question. It can be less convenient , but message passing is usually preferable to deadlock hell.

Like how Postfix works. That's a fun architecture to look at. Multiple processes and file based queue. Meanwhile I panic if I don't have PostgreSQL to save my data :/

Postgres doesn't use threads, it's a multiprocess architecture. Postfix probably does that on purpose to prevent losing outgoing (or incoming emails if you're doing POP3) in the event of a system crash/power loss.

Re: How fast are Linux pipes anyway? (2022)

#99

Earlier quoted context omitted.

Pipes are FIFO data buffers implemented in the kernel. For communication between threads of the same process, you can replace any pipe object by a userspace queue implementation protected by e.g. mutex + condition variable. It is functionally equivalent and has potential to be faster. And if you wrap all accesses in lock/unlock pairs (without locking any other objects in between) there is no danger of introducing any…

There are domain sockets if you need something more such as passing file descriptors. Both pipes and sockets (including TCP, with obvious limitations) can be done with zero copy given the right set of flags, thought things get harder if you have a complicated runtime (ie. garbage collection) involved. There's always explicitly mapped shared pages

[deleted]
Post reply on HN