Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Mighty Named Pipe

vincebuffalo.com

Post reply on HN
Print view View original

1–10 of 99 posts

Re: The Mighty Named Pipe

#2

Post by Malarkey73 »

Vince Buffalo is author of the best book on bioinformatics: Bioinformatics Data Skills (O'Reilly). It's worth a read for learning unix/bash style data science of any flavour.

Or even if you think you know unix/bash and data there are new and unexpected snippets every few pages that surprise you.

Re: The Mighty Named Pipe

#5

Post by aidos »

Nice article. Really easy to follow introduction.

I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit.

One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. 'Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these things exist....

Anyone know a good resource I should be using when I find myself in a situation like that?

Re: The Mighty Named Pipe

#6

Post by joosters »

frankerz wrote: post #4

How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) bigfile.gz vs tee bigfile.gz

It allows multiple, parallel pipes to each individual command, where the | allows just one.

Re: The Mighty Named Pipe

#7

Post by aidos »

frankerz wrote: post #4

How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) bigfile.gz vs tee bigfile.gz

In the tee case the substitution is actually going somewhere different than standard out (that's what tee does).

so:

    cmd1 | tee out.txt | cmd2
So tee is splitting the stream into two outputs, one that carries on out stdout (into cmd2) and the other one that is redirected into out.txt.

With process substitution you can do extra stuff on the way out, I guess (I've never seen it used for output before).

It looks like in the example given they're writing wc stuff to stderr while zipping the content (over stdout).

Nice to see that example, I hadn't even thought about the usefulness of process substitution for outputting like this!

Re: The Mighty Named Pipe

#8

Post by icebraining »

aidos wrote: post #5

Nice article. Really easy to follow introduction. I only discovered process substitution a few months ago but it's already become a frequently used tool in my kit. One thing that I find a little annoying about unix commands sometimes is how hard it can be to google for them. ' Unless you know to look for "Process Substitution" it can be hard to find information on these things. And that's once you even know these thi…

man pages!

  $ man bash
  /
Drops you right into the Process Substitution section.

Re: The Mighty Named Pipe

#9

Post by amelius »

If you like pipes, then you will love lazy evaluation. It is unfortunate, though, that Unix doesn't support that (operations can block when "writing" only, not when "nobody is reading").

Re: The Mighty Named Pipe

#10

Post by unhammer »

frankerz wrote: post #4

How does the > process substitution differ from simply piping the output with | ? For example (from Wikipedia) tee >(wc -l >&2) bigfile.gz vs tee bigfile.gz

Say that you have a program that splits its output into two files, each given by command line arguments. A normal run would be
but since the output is huge and your disk is old and dying, you want to run xz on it before saving it to disk, so use >():

    (xz - > out1.txt) -o2 >(xz - > out2.txt)
If you want to do several things in there, I recommend defining a function for clarity:

    pp () { sort -k2,3 -t$'\t' | xz - ; }
    (pp > out1.txt) -o2 >(pp > out2.txt)
Post reply on HN