Live data from Hacker News

The Mighty Named Pipe

vincebuffalo.com

11–20 of 99 posts

Re: The Mighty Named Pipe

#12
post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

man pages! $ man bash / Drops you right into the Process Substitution section.

    /
For those wondering, man, which uses less as a pager, has vi-like key bindings.

"/This is the origin of the regex literal syntax in most programming languages that have them. It was first introduced by Ken Thompson in the "ed" text editor.

Re: The Mighty Named Pipe

#13
post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

man pages! $ man bash / Drops you right into the Process Substitution section.

Ahhhh..haaa...ha....DOH! I've never even thought of looking at the manpage for bash before. Thanks, you've just made my life better.

Re: The Mighty Named Pipe

#14
Pipes are very cool and useful, but it's hard for me to understand this common worship of something like that. Yes, it's useful and elegant, but is it really the best thing since Jesus Christ?

Re: The Mighty Named Pipe

#15
post #4

How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) bigfile.gz vs tee bigfile.gz

When you connect to processes in a pipe such as ...

    a | b
you connect stdout (fd #1) of a to stdin (fd #0) of b. Technically, the shell process will create a pipe, which is two filedescriptors connected back to back. It then will fork two times (create two copies of itself) where it replaces standard output (filedescriptor 1) of the first copy by one end of the pipe and replaces standard input (filedescriptor 0) of the second copy by the other end of the pipe. Then the first copy will replace itself (exec) by a, the second copy will replace itself (exec) by b. Everything that a writes to stdout will appear on stdin of b.

But nothing prevents the shell from replacing any other filedescriptor by pipes. And when you create a subprocess by writing "" (symlink to /proc/self/fd/ which itself is a symlik to /proc//fd/), so that's what's replaced as a "name" to refer to the substituted process on "a"'s command line:

Try this:

    $ echo $$
    12345  # ( sort ) >( sort ) >( sort ) otherfile | sort
and in a second terminal

    $ pstree 12345 # 
If your system doesn't support the convenient /proc/self/fd/ shortcut, the shell might decide not to create a pipe, but rather create temporary fifos in /tmp and use those to connect the filedescriptors.

http://man7.org/linux/man-pages/man2/pipe.2.html

http://linux.die.net/man/2/dup

You can watch the syscalls as they are made:

    $ strace -fe fork,pipe,close,dup,dup2,execve bash -c 'tee 

Re: The Mighty Named Pipe

#17

Once you discover # avoid temporary files when some program needs two inputs: join -e0 -o0,1.1,2.1 -a1 -a2 -j2 -t$'\t' \

> # gawk doesn't care if it's given a regular file or the output fd of some process:

Something wonderful I found out the other day: Bash executes scripts as it parses them, so you can do all kinds of awful things. For starters,

    bash 
will have bash execute an infinite script that looks like

    echo hello
    echo hello
    echo hello
    ...
without trying to load the whole thing first.

After that, you can move onto having a script append to itself and whatever other dreadful things you can think of.

Re: The Mighty Named Pipe

#19
post #14

Pipes are very cool and useful, but it's hard for me to understand this common worship of something like that. Yes, it's useful and elegant, but is it really the best thing since Jesus Christ?

Maybe it's not the best thing since Jesus, but it's worth celebrating its birthday http://blog.fugue.it/2013-10-07-pipeday.html

Re: The Mighty Named Pipe

#20

Once you discover # avoid temporary files when some program needs two inputs: join -e0 -o0,1.1,2.1 -a1 -a2 -j2 -t$'\t' \

> # gawk doesn't care if it's given a regular file or the output fd of some process: Something wonderful I found out the other day: Bash executes scripts as it parses them, so you can do all kinds of awful things. For starters, bash will have bash execute an infinite script that looks like echo hello echo hello echo hello ... without trying to load the whole thing first. After that, you can move onto having a script…

That's actually one of the things that I really dislike with bash, that it doesn't read the whole script before executing it. I've been bitten by it before, when I write some long-running script, then e.g. write a comment at the top of it as it's running, and then when bash looks for the next command, it's shifted a bit and I get (at best) a syntax error and have to re-run :-(
Post reply on HN