GNU Parallel, where have you been all my life?
alexplescan.com
GNU Parallel, where have you been all my life?
1–10 of 277 posts
Re: GNU Parallel, where have you been all my life?
#2What about & and wait? Could it have been an adequate alternative?
Re: GNU Parallel, where have you been all my life?
#3GNU Parallel has been one of my go to tool to accomplish more on the terminal. Generate test data, transferring data from one node to another using rsync, run many-task, embarrassingly parallel jobs on HPC, pipelines with simple data dependencies but run over hundreds or files are some of the places where I use GNU Parallel.
Many thanks to Ole Tange for developing the wonderful tool and helping the users on Stack Overflow sites to this day.
Shameless plug, I am developing a tutorial on GNU Parallel to be presented at eScience conference in Cyprus this year: https://www.escience-conference.org/2023/tutorials/gnu_paral...
Re: GNU Parallel, where have you been all my life?
#4There's a shell script version of GNU parallel that's great for CI/CD pipeline tasks. You just keep it in your repo and source it as needed. It's incredibly useful, we use it in one build to batch process a few thousand things in groups of 25.
Edited to add: finally got signed in to work, you create the script via:
parallel --embed > scriptname.sh
Re: GNU Parallel, where have you been all my life?
#5What about & and wait? Could it have been an adequate alternative?
No, that is more messy and can easily leave lingering processes.
But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2...
GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.
Re: GNU Parallel, where have you been all my life?
#6What about & and wait? Could it have been an adequate alternative?
Probably for very simple use cases, but the real power in parallel really comes from the myriad of switches that enables so much more than what "&" and "wait" could do.
Here are a bunch of examples: https://www.gnu.org/software/parallel/parallel_examples.html
A fun one I end up using ~monthly or so for various things (usually with more switches added as needed):
GNU Parallel as queue system/batch manager
# start queue
true >jobqueue; tail -n+0 -f jobqueue | parallel
# add job
echo my_command my_arg >> jobqueue
# to start queue for remote execution
true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..Re: GNU Parallel, where have you been all my life?
#7xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.
Re: GNU Parallel, where have you been all my life?
#8Having a layer of parallelisation on top of good old sequential code seems like a very neat idea. It resolves headaches of learning how to run code in parallel in languages that aren’t necessarily my primary language (e.g. short, one-off scripts). Thanks for sharing!!
Re: GNU Parallel, where have you been all my life?
#9What about & and wait? Could it have been an adequate alternative?
You just taught me something
Re: GNU Parallel, where have you been all my life?
#10xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.
The entirety of GNU Parallel is just one Perl program. It could be copied over and used in a pinch. The installation itself is very simple and no special dependencies or privileges are needed.