Live data from Hacker News

How fast are Linux pipes anyway? (2022)

mazzo.li

101–110 of 116 posts

Re: How fast are Linux pipes anyway? (2022)

#101
post #56

Earlier quoted context omitted.

Not necessarily. Good comments go a long way.

I have a very long list of things that were good, worked well, and ended up rejected because the team didn't want to put in effort to learn how they work. My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but bei…

That's a slightly different scenario. If your product was written in C and your boss was disappointed that it couldn't saturate a 40Gbps link, a well—written and commented implementation (in C) that could would probably not be rejected.

As others have pointed out, a lot of your failures sound more political than technical. Dealing with idiots and jumping through political hoops to get them to go along with an idea that they didn't think up themselves can be extremely tiring. I cannot claim any particular expertise in that area myself, and I can't think of any way around it (aside from working alone, of course). I figure cooperation is the cost of progress.

Re: How fast are Linux pipes anyway? (2022)

#102

Earlier quoted context omitted.

wouldnt it have been easier to port the Haxe code to AS3? afaik both language are very similar. Then you wouldn't have a problem and you wouldnt have to rely on them to learn your codebase. Likewise I doubt that the parser being written in C was the problem, the problem was probably that (I assume?) you didnt also provide the convenient python bindings. Otherwise it's hard to believe that someone would throw away fin…

Porting HaXe to AS3 wouldn't have solved the C++ part of the server component. Even if I didn't need the C++ part, HaXe way of doing things is very different from Flex. Also, the code for SWF generation required quite a bit of linear algebra, which scared the original developers (the code had to take various properties of display objects and convert them into matrices of affine transformations, because that's how SWF…

well from your response I think I can guess why you keep running into these problems so often. While superficially you understand that people don't want to do the extra work you don't really understand why nor do you even understand what their problem is with e.g Prolog, Haxe or C. The complex interconnecting reasons around popularity, IDE support, performance, ecosystem, hiring, maintenance burden, security issues are instantly obvious to someone who does the minimum amount of research to understand what is genuinely required of a modern language. The term "old" or "modern" should obviously be understood as a catchall term to make knowledgable people instantly infer that an entire list of ecosystem related things such as e.g package management, IDE or libraries haven't progressed for X, in this case Prolog. Regarding the IDE you don't even list the debugger, it sounds like you literally downloaded and worked with the swi prolog CLI and you were fine with this. Other people are not fine with this because they actually need to work, maintain and expand on this code and they can easily do this with the existing language but can't do this with yours. They don't want to relearn an entire ecosystem that is broken in hundreds if not thousands of ways just because you were able to technically write one console program in a notepad in another language. Obviously the state of the art is not to work with a raw text editor or emacs like it's literally the 80s and do printf debugging. Then add to all of that your disposition to quickly consider people as lazy and ignorant and it makes it clear that they won't even be able to rely on you to fix any future issues because who knows what kind of ego problem will come up that prevents you from just dealing with an issue. If you genuinely don't understand the gigantic editor support difference between Python and Prolog then you are the one that is lazy and it's up to you to change that. I am not lazy for instantly dismissing prolog from one look at their swi prolog website and a tiny google search because I have the experience to understand all these factors as a whole. One look at the swi-prolog website, the prolog syntax and lack of VS or intellij support is enough to know that this language isn't going to be relevant again. Ada actually improved their website and documentation a lot when I pointed the same out to them which was a big surprise because most people don't care they just keep going in the wrong way. Ada still doesn't have a good IDE though and their IDE is far better than what Prolog seems to have.

Re: How fast are Linux pipes anyway? (2022)

#103
post #56

Earlier quoted context omitted.

Not necessarily. Good comments go a long way.

I have a very long list of things that were good, worked well, and ended up rejected because the team didn't want to put in effort to learn how they work. My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but bei…

The part you left out is where you inevitably leave the company and now there is this bespoke snowflake which nobody knows about doing god knows what transpiling stuff under the hood. With the average tenure of a software engineer being a few years maintainability and standardization are important above all. This is one of the reasons why opinionated and simple languages like golang have gained large popularity.

I do agree that genuine software excellence and craft is rare in corporate settings. If you want to make a piece of excellent software like curl or sqlite you will be doing it on your own, and you most likely will not be making a living off of it either.

Re: How fast are Linux pipes anyway? (2022)

#104
post #22
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

That may have been surprising, but, if you think about it a little deeper, it makes perfect sense. Programs in a pipeline execute concurrently. If they didn’t, pipelines wouldn’t be useful. For instance a pipeline that downloads a tar file with curl and then untars it. If you wait for curl to finish before running tar, you run in to all sorts of problems. For instance, where do you store the intermediate tar file if…

Powershell implements pipelines deterministically and without concurrency, and you can be very precise about it. Of course, it will use OS pipes if you include binaries in your pipeline.

Nushell looks like it also has an internal implementation of pipelines. But I can't read rust so that's just my assumption.

Re: How fast are Linux pipes anyway? (2022)

#105
post #4

One surprising fact about Linux pipes I stumbled across 4 years ago is that using a pipe can create indeterministic behavior: https://www.gibney.org/the_output_of_linux_pipes_can_be_inde...

I'm genuinely curious, how else could this work? It's like spawning threads, it's inherently indeterministic.

My shell throws an error if I try to pipe to a command that doesn't accept piped input. It's just better design.

This is also why python sucks - if you feed it garbage, the error may surface a long way away and it may do a lot of damage while it's underwater

Re: How fast are Linux pipes anyway? (2022)

#106
post #104
post #22

Earlier quoted context omitted.

That may have been surprising, but, if you think about it a little deeper, it makes perfect sense. Programs in a pipeline execute concurrently. If they didn’t, pipelines wouldn’t be useful. For instance a pipeline that downloads a tar file with curl and then untars it. If you wait for curl to finish before running tar, you run in to all sorts of problems. For instance, where do you store the intermediate tar file if…

Powershell implements pipelines deterministically and without concurrency, and you can be very precise about it. Of course, it will use OS pipes if you include binaries in your pipeline. Nushell looks like it also has an internal implementation of pipelines. But I can't read rust so that's just my assumption.

What do you mean “without concurrency”? One program runs entirely before the other starts?

Re: How fast are Linux pipes anyway? (2022)

#107
post #104

Earlier quoted context omitted.

Powershell implements pipelines deterministically and without concurrency, and you can be very precise about it. Of course, it will use OS pipes if you include binaries in your pipeline. Nushell looks like it also has an internal implementation of pipelines. But I can't read rust so that's just my assumption.

What do you mean “without concurrency”? One program runs entirely before the other starts?

Powershell pipelines are an engine construct rather than OS pipes or file descriptors. (If you include OS binaries in a PS pipeline, it will map the internal pipeline to OS pipes for that element of the pipeline, of course.)

Every Powershell command has a begin, process, and end block. (If you don't write these explicitly, your code goes in an implicit end block.)

When a pipeline is evaluated:

1. From left to right, the begin block of each command is run, sequentially. No process or end blocks are run until every begin block has run.

2. Each command's process block is run, once per object piped in. A process block can output zero, one or many objects; I'd have to check on a computer, but IIRC this is "breadth-first" - each object that a process block outputs is passed to the next process block before returning control to the current process block.

3. After all process blocks are exhausted, from left to right, each command's end block is run. Commands that did not declare a process block receive all piped objects as a single collection. Any output from the end block triggers the process block to the right.

4. When all end blocks have completed, the pipeline is stopped

5. Errors in Powershell can be terminating or non-terminating. When a terminating error is thrown, the pipeline is stopped

6. There is a special StopPipeline error which stops the pipeline but is handled by the engine so the user never sees it. That's how `select -First 5` works (for PS `select`, not gnu select).

Pipelines only operate on streams 0 and 1, as with OS pipes. The other streams (ps has 7) are handled immediately, modulo some buffering behaviour intoxicated for performance reasons. Broadly speaking, the alternate streams are suppressed or enabled by defaults and by switches on each command individually and are rendered by the engine and given to the console to display. But they can also be redirected or captured in variables.

You can do asynchrony in Powershell; threading is offered by a construct called "runspaces". These are not inherently connected to the pipeline, but pipelined commands can implement them, e.g. `foreach -Parallel {do-stuff}`

Re: How fast are Linux pipes anyway? (2022)

#108
post #107

Earlier quoted context omitted.

What do you mean “without concurrency”? One program runs entirely before the other starts?

Powershell pipelines are an engine construct rather than OS pipes or file descriptors. (If you include OS binaries in a PS pipeline, it will map the internal pipeline to OS pipes for that element of the pipeline, of course.) Every Powershell command has a begin, process, and end block. (If you don't write these explicitly, your code goes in an implicit end block.) When a pipeline is evaluated: 1. From left to right,…

Ok, so it sounds like Powershell would have the exact same issue as the Linux pipes. The issue has nothing to do with determinism with the pipeline construction and everything to do with the fact that part of the pipeline writes to stderr, which you could call stream 2.

Re: How fast are Linux pipes anyway? (2022)

#109
post #49

So if I understand correctly, vmsplice is more of a mini shared memory mechanism between two processes, if used on both the reader and writer end simultaneously? Meaning both processes need to be exceptionally careful in when they read and write to the buffers and how it is returned after use. Hot, yet scary at the same time. Other main takeaway, it’s a bit sad that the naive implementation everybody will write, is 2…

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

Which can be the case. Saving 15ms a few times a week/day/hour(?) vs hours of developer maintenance time over the life if the thing can still be an issue.

Re: How fast are Linux pipes anyway? (2022)

#110

Earlier quoted context omitted.

And if you try to write the 20x faster version, your coworkers will think you are over-complicating and not being a team player.

In the end they'll just use a Lambda.

Is that beneficial or not? I'm still too much of a novice to know.
Post reply on HN