Didn't know about the book: https://zenodo.org/record/1146014 (discussed 4 years back: https://news.ycombinator.com/item?id=20726631)
See also https://hn.algolia.com/?q=gnu+parallel for other related discussions.
GNU Parallel, where have you been all my life?
11–20 of 277 posts
Re: GNU Parallel, where have you been all my life?
#12What about & and wait? Could it have been an adequate alternative?
No, that is more messy and can easily leave lingering processes. But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2... GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.
GNU parallel supports bash functions, provided you "export -f" them beforehand
Re: GNU Parallel, where have you been all my life?
#13xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.
Last time I checked (which was a few years ago, admittedly), some popular ystem's xargs were too old to support parallelism -- Mac in particular.
Re: GNU Parallel, where have you been all my life?
#14Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick:
find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}Re: GNU Parallel, where have you been all my life?
#15GNU Parallel has been one of my go to tool to accomplish more on the terminal. Generate test data, transferring data from one node to another using rsync, run many-task, embarrassingly parallel jobs on HPC, pipelines with simple data dependencies but run over hundreds or files are some of the places where I use GNU Parallel. Many thanks to Ole Tange for developing the wonderful tool and helping the users on Stack Ove…
I'm surprised the CPU would in any way be the bottleneck for transferring data. Is it really faster to parallelize that?
Re: GNU Parallel, where have you been all my life?
#16I much prefer rush over parallel. Namely that everything is executed as a bash shell.
Re: GNU Parallel, where have you been all my life?
#17Wonderful... Thanks for sharing.
Re: GNU Parallel, where have you been all my life?
#18Learning Parallel pays high dividends for the rest of your life.
Re: GNU Parallel, where have you been all my life?
#19It is sort if a shame that tools can’t figure out how to parallelize things without being herded like cattle to do so.
It might be a culture thing. In .NET code I see people running things in parallel a lot within code but maybe this is less so for linux tools.
Maybe functional programming style could lend to a parallel-first programming style, with heuristics to decide when it isn’t worth it.
Re: GNU Parallel, where have you been all my life?
#20Someone gifted an old blade server to me a few years ago. Very slow, but 16 cores and 24 gig of RAM. At the time I was making a lot of video art with ffmpeg, without a GPU. That version of ffmpeg wasn't optimized for multiple cores so rendering was really slow and sequential. I discovered Parallel and set the server to process large videos with most of the cores in parallel. Voila, it chewed through a massive amount of media fairly quickly. Faster than the hard drives actually.