"Do one thing and do it well"
GNU Parallel, where have you been all my life?
21–30 of 277 posts
Re: GNU Parallel, where have you been all my life?
#22xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.
See my comment above, there's a shell version you can store in your project repository and use wherever you want with zero installation!
Re: GNU Parallel, where have you been all my life?
#23Folks who are here and interested in parallelization for CI/CD may also be interested in Dagger.io — I had heard about it on HN over the years but not played w it. It's basically a more fine-grained Docker-like executor with better caching and utilities for spinning up services and running tests.
Curious if anyone else has experiences with it, honestly been surprised at how little I've heard about it
Re: GNU Parallel, where have you been all my life?
#24parallel is great but its default behaviors never quite seem to match my needs, so every time I use it I have to spend some time consulting the man page. Fortunately, the man page is more than up to the task.
But because of the mini learning curve on each use and because I find I need a little more boiler plate to use parallel, I use xargs -P more often, only using parallel when I need its special features (e.g. multiple hosts or collating the output streams).
Oh also, parallel itself can be a bit of a resource hog. (Obviously that depends a lot on how you're using it-- but I mean in cases where xargs' usage is unnoticeable I sometimes have to change the size of my jobs to get parallel out of the way).
Re: GNU Parallel, where have you been all my life?
#25xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.
Indeed, xargs can be a better option, but it has trouble doing some tasks efficiently.
For example, translating a large list of IPv4 ranges into a standard format for a firewall rule-set parser:
cat ~/blacklist.p2p | parallel --ungroup --eta --jobs 20 "ipcalc {} | sed '2!d' " | grep -Ev '^(0.|255.|127.)' >> ~/blacklist_p2p_converted
Makes an annoyingly slow task tolerable, as parallel doesn't block while fetching to preserve order. We probably should rewrite this to be more efficient, but this task is run infrequently.
Happy computing =)
Re: GNU Parallel, where have you been all my life?
#26GNU parallel is great for the kind of tasks highlighted in the post. Note that being written in Perl, it's slower than its simpler C counterpart moreutils parallel. And that in many uses cases xargs --max-procs=$(nproc) can replace it.
Re: GNU Parallel, where have you been all my life?
#27GNU Parallel has been one of my go to tool to accomplish more on the terminal. Generate test data, transferring data from one node to another using rsync, run many-task, embarrassingly parallel jobs on HPC, pipelines with simple data dependencies but run over hundreds or files are some of the places where I use GNU Parallel. Many thanks to Ole Tange for developing the wonderful tool and helping the users on Stack Ove…
I'm surprised the CPU would in any way be the bottleneck for transferring data. Is it really faster to parallelize that?
[deleted]
Re: GNU Parallel, where have you been all my life?
#28GNU Parallel has been one of my go to tool to accomplish more on the terminal. Generate test data, transferring data from one node to another using rsync, run many-task, embarrassingly parallel jobs on HPC, pipelines with simple data dependencies but run over hundreds or files are some of the places where I use GNU Parallel. Many thanks to Ole Tange for developing the wonderful tool and helping the users on Stack Ove…
I'm surprised the CPU would in any way be the bottleneck for transferring data. Is it really faster to parallelize that?
It's more GNU Parallel has host groups in a config so you can send files for a job to the right one where its going to execute and bring things back. Essentially it can turn a local xargs type job into any kind of remote task execution including dealing with files locally needing to be remote.
Re: GNU Parallel, where have you been all my life?
#29Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick: find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}
Any particular reason to use -print0 and pipe instead of -exec?
Re: GNU Parallel, where have you been all my life?
#30[deleted]