Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel – The command line power tool

slideshare.net

Post reply on HN
Print view View original

1–10 of 29 posts

Re: GNU Parallel – The command line power tool

#3

Post by oofabz »

I use GNU Parallel. I like it because its interface is simple - input is piping filenames to it, just like xargs, and output is nicely collated to the screen.

I used to use ppss, which does the core task just as well, but the interface is more complex.

I mostly use these tools to optimize large numbers of PNGs before deployment, using optipng, pngout, and/or my own lossypng. These programs take a while to run so using all my cores gets the job done a lot quicker.

Re: GNU Parallel – The command line power tool

#4

Post by shrike »

I use GNU Parallel with s3cmd to move big data sets in and out of S3. I can easily saturate any network connection. I was able to GET ~2TB from S3 onto a Gluster cluster in a little more than an hour by using GNU Parallel to spread the GETs across 8 instances. Incredibly powerful, easy to use tool.

Re: GNU Parallel – The command line power tool

#6

Post by dredmorbius »

rcthompson wrote: post #2

i use GNU parallel exclusively in place of xargs simply because it has --dry-run.

Nice. I just had to check to see if xargs has a similar feature. It doesn't, though the --no-run-if-empty and --verbose options are both handy. I believe I've used xargs with "echo " to proof output before committing it.

You can simply re-run that piped to bash (or your shell of choice) to execute commands if you wish (say, if parallel isn't available).

E.g.,

     echo foo bar baz | xargs -n1 -t echo ls | bash
... will execute 'ls foo; ls bar; ls baz', while showing the expanded command.

Re: GNU Parallel – The command line power tool

#7

Post by ealloc »

I use parallel all the time for embarrasingly parallel scientific computations on a cluster. It is very easy to use and elegant, and it's one of the programs I'm most grateful for.

Recently the developers fixed a major bug for me, that child jobs on other nodes would not be killed when parallel was killed. This was the only thing stopping me from recommending it to my labmates, now there's no reason not to use it!

Re: GNU Parallel – The command line power tool

#9

Post by jasomill »

_ZeD_ wrote: post #5
rcthompson wrote: post #2

i use GNU parallel exclusively in place of xargs simply because it has --dry-run.

wouldn't an "echo" just do the same? $ mkdir a $ cd a $ touch b c d e $ find -type f | xargs echo rm rm ./b ./d ./e ./c $ ls b c d e $ find -type f | xargs rm $ ls $

One advantage of "parallel --dry-run" over "xargs echo" is that the former quotes its output:

    $ touch 'Ham
    Jam
    Spam'
    $ touch 'J.R. "Bob" Dobbs'
    $ find . -type f -print0 | parallel -n1 -0 --dry-run echo
    echo ./Ham'
    'Jam'
    'Spam
    echo ./J.R.\ \"Bob\"\ Dobbs
    $ find . -type f -print0 | xargs -n1 -0 echo echo
    echo ./Ham
    Jam
    Spam
    echo ./J.R. "Bob" Dobbs
For my own "dry runs", though, I've always preferred passing the command line to

    #include 
    
    int main(int argc, char* argv[]) {
        char** argp;
        int i;
        printf("argc = %d\n", argc);
        for (i = 0, argp = argv; *argp != 0; ++argp, ++i) {
            printf("argv[%d] = %s\n", i, *argp);
        }
        return 0;
    }
to remove all reasonable doubt.
Post reply on HN