GNU Parallel Cheat Sheet [pdf]

A few slightly more advanced GNU Parallel features that I've used:

- --joblog writes out a detailed logfile of the jobs, which can be used to resume from interrupted runs with --resume{,-failed}

- `--slf filename` can be used to provide a list of ssh logins to remote worker nodes to run jobs. Importantly, parallel will automatically reread this list when it changes. This lets you very easily distribute batch jobs across preemptible gcloud vms (or ec2 spot instances) and gracefully handle worker nodes appearing/disappearing with just a few lines of bash https://gist.github.com/gpittarelli/5e14fb772ce0230a3c40ffad...

- When used with bash, parallel can run bash functions if you export them with `export -f functionName` .

Parallel is Good Stuff (tm) and works very well but I haven't had much cause to use it.

For ad-hoc system modifications I've found myself using tmux's synchronize-panes feature, or xargs. For anything bigger or more involved then I break out Ansible/Chef/Puppet depending on which client project I'm working on.

I remember one place I worked at had a huge elaborate configuration/deployment system hand written by the head IT guy which used Parallel+bash+perl extensively. Thing is, while it was a great system, I could make the same changes in Ansible or Puppet with a couple of lines and push them within minutes, while making changes using the hand written system might take hours. Plus no logging and poor error handling led to all sorts of problems with that system, despite it being a real labour of love by that wacky Finnish dude.

However this sheet is really nice because it is just one side of a letter/A4 piece of paper and lays out the information clearly. I definitely want to mess around with Parallel now because of this cheat sheet. I wonder how it was typeset or laid out on the page? I try to write my own cheat sheets but they always seem way too sparse with too much white space. Maybe it is written in LaTeX or similar.

Those are all really good tips, thank you for sharing them.

It has some more granular control over "pasting" in values. For example, you can use {} for the arg value itself, or you can use {.} for just what's before the extension, or you can use {/} for the basename, or {/.} for the basename without extension, etc. You can also get progress bar, ETA, etc.

parallel is like xargs++; for simple cases it does the same thing as xargs, but it also has many more advanced features such as:

- Splitting input lines into multiple fields and building more complex commands from them

- Running jobs on remote nodes

- Pausing/resuming batch jobs (--joblog)

- ETA and progress bars

- Passing data to programs on stdin and generally many, many other ways of distributing and collecting data that xargs can't do

You can see a bunch of examples at: https://www.gnu.org/software/parallel/man.html

  $ PAGER=cat man xargs | wc -l
  259
  $ PAGER=cat man parallel | wc -l
  3985