Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Mighty Named Pipe

vincebuffalo.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: The Mighty Named Pipe

#91

Post by JoeAltmaier »

valarauca1 wrote: post #75
amelius wrote: post #66

Earlier quoted context omitted.

Well, in any case, the problem can be resolved by adding a kernel-level api function that allows one to wait (block) until results are requested from the other end of the pipe.

Why? The opposite is already true and has the same effect. Each stage of the pipeline is executed when it has data to execute. So ultimately the main blocking event is IO (normally the first stage in a pipeline). Every other process is automatically marked as blocked, until its stdin is populated by the output of the former. Once its task is complete it re-checks stdin, and if nothing is present blocks itself. So the…

There are arguments for supply-driven processing, AND for demand-driven. It all comes down to latency and speed arguments.

Re: The Mighty Named Pipe

#92

Post by AceJohnny2 »

Filligree wrote: post #23
aidos wrote: post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

For this in particular, try the Advanced Bash Scripting doc, http://www.tldp.org/LDP/abs/html/ There's a bunch of interesting constructs there. Most of them also apply to improved shells such as zsh, though some are just pointless there.

I've seen the ABS guide criticized for being obsolete and recommending wrong or obsolete best practices. A recommended replacement is The Bash Hacker's Wiki: http://wiki.bash-hackers.org/doku.php

Which itself recommends as best (current) alternative: Greg's Bash Guide: http://mywiki.wooledge.org/BashGuide

Re: The Mighty Named Pipe

#93

Post by aidos »

david-given wrote: post #68
aidos wrote: post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

Be aware that process substitution (and named pipes) can bite you in the arse in some situations --- for example, if the program expects to be able to seek in the file. Pipes don't support this and the program will see it as an I/O error. This'd be fine if programs just errored out cleanly but they frequently don't check that seeking succeeds. unzip treats a named pipe as a corrupt zipfile, for example: $ unzip

Useful. I'm assuming those are cases where normal pipes would fail too? So you can't do:

    cat z | unzip   # I know, uuoc, demo only
It's just with the process substitution you have more flexibility to shoot yourself in the foot?

Re: The Mighty Named Pipe

#94

Post by frankerz »

anateus wrote: post #84

In fish shell the canonical example is this: diff (sort a.txt|psub) (sort b.txt|psub) The psub command performs the process substitution.

It seems like fish shell's ">" process substitution equivalence is not working as well as bash's though

https://github.com/fish-shell/fish-shell/issues/1786

Re: The Mighty Named Pipe

#95

Post by wtbob »

vertex-four wrote: post #52
omaranto wrote: post #42

Earlier quoted context omitted.

That's weird, isn't it? You wanted to know how to use a feature of bash and didn't check the manual?

Do you expect `man python` to output the full reference for the Python programming language?

I wish that it did. The man pages really are supposed to be the manual.

Re: The Mighty Named Pipe

#96

Post by david-given »

aidos wrote: post #93
david-given wrote: post #68

Earlier quoted context omitted.

Be aware that process substitution (and named pipes) can bite you in the arse in some situations --- for example, if the program expects to be able to seek in the file. Pipes don't support this and the program will see it as an I/O error. This'd be fine if programs just errored out cleanly but they frequently don't check that seeking succeeds. unzip treats a named pipe as a corrupt zipfile, for example: $ unzip

Useful. I'm assuming those are cases where normal pipes would fail too? So you can't do: cat z | unzip # I know, uuoc, demo only It's just with the process substitution you have more flexibility to shoot yourself in the foot?

stdin and stderr are assumed to always be streams, so anything which accepts them won't seek on them.

However, if you give a program a filename on a command line, the program's likely to assume it's an actual file.

Re: The Mighty Named Pipe

#98

Post by Wicher »

aidos wrote: post #93
david-given wrote: post #68

Earlier quoted context omitted.

Be aware that process substitution (and named pipes) can bite you in the arse in some situations --- for example, if the program expects to be able to seek in the file. Pipes don't support this and the program will see it as an I/O error. This'd be fine if programs just errored out cleanly but they frequently don't check that seeking succeeds. unzip treats a named pipe as a corrupt zipfile, for example: $ unzip

Useful. I'm assuming those are cases where normal pipes would fail too? So you can't do: cat z | unzip # I know, uuoc, demo only It's just with the process substitution you have more flexibility to shoot yourself in the foot?

Aside: If I recall correctly, with the zip file format, the index is at the end of the file. A (named) pipe works fine with, for instance, a bzipped tarball.

I wouldn't be surprised if the ZIP file format has its origins outside the Unix world given its pipe-unfriendlyness.

Re: The Mighty Named Pipe

#99

Post by aidos »

dandelany wrote: post #62
aidos wrote: post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

I often use SymbolHound in these cases - it's a search engine with support for special characters; for example: http://symbolhound.com/?q=%3C%28%29

Thanks for that. I've already made use of it since you pointed it out.
Post reply on HN