Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel – The command line power tool

slideshare.net

Post reply on HN
Print view View original

21–29 of 29 posts

Re: GNU Parallel – The command line power tool

#21

Post by dredmorbius »

rcthompson wrote: post #12
dredmorbius wrote: post #6

Earlier quoted context omitted.

Nice. I just had to check to see if xargs has a similar feature. It doesn't, though the --no-run-if-empty and --verbose options are both handy. I believe I've used xargs with "echo " to proof output before committing it. You can simply re-run that piped to bash (or your shell of choice) to execute commands if you wish (say, if parallel isn't available). E.g., echo foo bar baz | xargs -n1 -t echo ls | bash ... will ex…

Yes, you can use echo, but that won't work if the command is something like "ls DIR | wc -l".

Depending on how you want to do your counts or piping, you could run that after the xargs / parallel execution, which would be much more efficient (fewer processes and execs) anyhow.

Re: GNU Parallel – The command line power tool

#22

Post by seryoiupfurds »

peterwwillis wrote: post #20
jasomill wrote: post #9

Earlier quoted context omitted.

One advantage of "parallel --dry-run" over "xargs echo" is that the former quotes its output: $ touch 'Ham Jam Spam' $ touch 'J.R. "Bob" Dobbs' $ find . -type f -print0 | parallel -n1 -0 --dry-run echo echo ./Ham' 'Jam' 'Spam echo ./J.R.\ \"Bob\"\ Dobbs $ find . -type f -print0 | xargs -n1 -0 echo echo echo ./Ham Jam Spam echo ./J.R. "Bob" Dobbs For my own "dry runs", though, I've always preferred passing the command…

Why does the parallel example look weird? It put the single quotes in all the wrong places, and completely around Jam. It should have printed echo './Ham Jam Spam' but your output looks different. As an example of how it should look, try this: $ find . -type f -print0 | xargs -n1 -0 perl -e'print "\"$_\" " for (@ARGV)' "./Ham Jam Spam"

It looks like it's only single-quoting the characters that need it, in his example the newlines.

Re: GNU Parallel – The command line power tool

#23

Post by davvolun »

peterwwillis wrote: post #20
jasomill wrote: post #9

Earlier quoted context omitted.

One advantage of "parallel --dry-run" over "xargs echo" is that the former quotes its output: $ touch 'Ham Jam Spam' $ touch 'J.R. "Bob" Dobbs' $ find . -type f -print0 | parallel -n1 -0 --dry-run echo echo ./Ham' 'Jam' 'Spam echo ./J.R.\ \"Bob\"\ Dobbs $ find . -type f -print0 | xargs -n1 -0 echo echo echo ./Ham Jam Spam echo ./J.R. "Bob" Dobbs For my own "dry runs", though, I've always preferred passing the command…

Why does the parallel example look weird? It put the single quotes in all the wrong places, and completely around Jam. It should have printed echo './Ham Jam Spam' but your output looks different. As an example of how it should look, try this: $ find . -type f -print0 | xargs -n1 -0 perl -e'print "\"$_\" " for (@ARGV)' "./Ham Jam Spam"

Having never used parallel, I still believe parallel was correct.

./Ham' 'Jam' 'Spam

would be identical to ./Ham\nJam\nSpam (if \n were the correct translation to the newline in this case) or './Ham Jam Spam'

This would be identical to what you wrote, but only punts to quotes when it doesn't have a canonical method of representing the character otherwise. The fact that you don't need to explicitly concatenate two strings in the shell may be what's throwing you off?

Interestingly enough, 'Ham\n\nJam\nSpam' becomes

./Ham' '' 'Jam' 'Spam

So parallel is just literally outputting all newlines using quotes. I believe this would be identical, if you analyzed it and saw that two newlines are next to each other:

./Ham'

'Spam' 'Jam

Re: GNU Parallel – The command line power tool

#24

Post by gnoe »

Is parallel buggy or is it just me? For example if i have a list of ip addresses:

  $ cat ips.txt | sort | uniq -c | sort -rn

   3 127.0.0.1
   2 192.168.1.1
   1 192.168.1.2
Now i want to reformat the output of uniq -c, i want the count to the last column:

  $ cat ips.txt | sort | uniq -c | sort -rn | parallel --colsep ' ' echo {2} {1}
But gives empty output.. what gives? It only works if I double pipe it thru parallel like this:

  $ cat ips.txt | sort | uniq -c | sort -rn | \
      parallel --trim lr echo | parallel --colsep ' ' echo {2} {1}

  127.0.0.1 3
  192.168.1.1 2
  192.168.1.2 1

Re: GNU Parallel – The command line power tool

#28

Post by ole_tange »

gnoe wrote: post #24

Is parallel buggy or is it just me? For example if i have a list of ip addresses: $ cat ips.txt | sort | uniq -c | sort -rn 3 127.0.0.1 2 192.168.1.1 1 192.168.1.2 Now i want to reformat the output of uniq -c, i want the count to the last column: $ cat ips.txt | sort | uniq -c | sort -rn | parallel --colsep ' ' echo {2} {1} But gives empty output.. what gives? It only works if I double pipe it thru parallel like this…

You have more than 1 space from uniq. 2 options:

  parallel --colsep ' +' echo {2} {3}
or:

  parallel --colsep ' ' echo {7} {8}

Re: GNU Parallel – The command line power tool

#29

Post by gnoe »

ole_tange wrote: post #28
gnoe wrote: post #24

Is parallel buggy or is it just me? For example if i have a list of ip addresses: $ cat ips.txt | sort | uniq -c | sort -rn 3 127.0.0.1 2 192.168.1.1 1 192.168.1.2 Now i want to reformat the output of uniq -c, i want the count to the last column: $ cat ips.txt | sort | uniq -c | sort -rn | parallel --colsep ' ' echo {2} {1} But gives empty output.. what gives? It only works if I double pipe it thru parallel like this…

You have more than 1 space from uniq. 2 options: parallel --colsep ' +' echo {2} {3} or: parallel --colsep ' ' echo {7} {8}

Thanks!, but how come the whitespace is not trimmed by --trim lr? The manpage says it trims whitespace left and right if --colsep is used.
Post reply on HN