Design of GNU Parallel (2015)
1–10 of 75 posts
Re: Design of GNU Parallel (2015)
#2I couldn’t make heads or tails of what this would be useful for from the OP (maybe it’s something I should already have known), but this from the official site was pretty helpful: https://www.gnu.org/software/parallel/parallel_cheat.pdf
Re: Design of GNU Parallel (2015)
#3parallel is a tool I've reached for many times; the citation bit it prints is odd - it seems to assume that the general use case is research/academic - but easily squelched.
A sample use case would be having a file that has words in it, one per line, and you want to run a program that operates on each word (device name, dollar amount, whatever). Sure, you can use a loop, but if the words and actions are independent, parallel is one way to spin up N copies of your program and pass it a single word from the file. Can get around Python's GIL without having to use multiprocessing or threads (as a more concrete example).
Didn't realise that it busy waits, but I'm typically running it on a not very busy server with tens of cores.
Re: Design of GNU Parallel (2015)
#4I found GNU parallel useful when I wanted to queue up transcoding of flac files to mp3 on my Raspberry Pi. A few ffmpeg flags plus a list of files meant I could easily just saturate one job per core with a one-line bash command.
Re: Design of GNU Parallel (2015)
#5I found GNU parallel useful when I wanted to queue up transcoding of flac files to mp3 on my Raspberry Pi. A few ffmpeg flags plus a list of files meant I could easily just saturate one job per core with a one-line bash command.
I've used it to parallelise updating hundreds of helm releases whose CI pipelines had ceased to exist. It is a neat tool.
Re: Design of GNU Parallel (2015)
#6parallel is a tool I've reached for many times; the citation bit it prints is odd - it seems to assume that the general use case is research/academic - but easily squelched. A sample use case would be having a file that has words in it, one per line, and you want to run a program that operates on each word (device name, dollar amount, whatever). Sure, you can use a loop, but if the words and actions are independent,…
Thankfully both Debian and Arch patch out the citation nonsense.
Re: Design of GNU Parallel (2015)
#7Wait what: `parallel` is a Perl script!? [1]
I would have thought it's black magic with assembler optimisations for MIPS and special considerations for HP-UX...
This is such a lovely and interesting writeup, it's wonderful that people take their time to share so generously!
[1] : an 11k loc petal script, you can read along here: https://github.com/gitGNU/gnu_parallel/blob/master/src/paral...
Re: Design of GNU Parallel (2015)
#8I couldn’t make heads or tails of what this would be useful for from the OP (maybe it’s something I should already have known), but this from the official site was pretty helpful: https://www.gnu.org/software/parallel/parallel_cheat.pdf
That cheat sheet is super enlightening!
But quite useless as it'll print poorly and is overall a waste of resources to have that lovely beach scene in the background.
Re: Design of GNU Parallel (2015)
#9I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications.
And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start another one if needed.
Re: Design of GNU Parallel (2015)
#10I couldn’t make heads or tails of what this would be useful for from the OP (maybe it’s something I should already have known), but this from the official site was pretty helpful: https://www.gnu.org/software/parallel/parallel_cheat.pdf
That cheat sheet is super enlightening! But quite useless as it'll print poorly and is overall a waste of resources to have that lovely beach scene in the background.
Try this resource instead. Although it is 100 pages, the introductory part is already useful in and of itself!
https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pd...