Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

31–40 of 277 posts

Re: GNU Parallel, where have you been all my life?

#31

Post by adrian_b »

SPBS wrote: post #7

xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.

GNU Parallel has been created precisely for solving some deficiencies of xargs.

While there are cases when it makes sense to stick to what is specified by POSIX, there are also cases when the POSIX specification is so obsolete that using POSIX instead of some free ubiquitous programs is a big mistake.

Among these latter cases are writing scripts for a POSIX shell instead of writing them for bash and using xargs instead of parallel.

Re: GNU Parallel, where have you been all my life?

#32

Post by em500 »

ketanmaheshwari wrote: post #10
SPBS wrote: post #7

xargs is more useful because it's posix so you can always guarantee it to be there (whereas with GNU Parallel you probably have to reach for a package manager to install it first). The ergonomics are worse though, as usual.

The entirety of GNU Parallel is just one Perl program. It could be copied over and used in a pinch. The installation itself is very simple and no special dependencies or privileges are needed.

Except Perl isn't always present by default either (e.g. in Arch Linux or FreeBSD).

Re: GNU Parallel, where have you been all my life?

#33

Post by throwaway277432 »

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL?

And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers.

These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need it to.

Re: GNU Parallel, where have you been all my life?

#36

Post by Gabrys1 »

asicsp wrote: post #29
ssddanbrown wrote: post #14

Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick: find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}

Any particular reason to use -print0 and pipe instead of -exec?

-exec would not be parallel, pipe to parallel makes it parallel

Re: GNU Parallel, where have you been all my life?

#37

Post by [deleted] »

bloopernova wrote: post #4

There's a shell script version of GNU parallel that's great for CI/CD pipeline tasks. You just keep it in your repo and source it as needed. It's incredibly useful, we use it in one build to batch process a few thousand things in groups of 25. Edited to add: finally got signed in to work, you create the script via: parallel --embed > scriptname.sh It's about 14,000 lines of awesome and works on "ash, bash, dash, ksh,…

[deleted]

Re: GNU Parallel, where have you been all my life?

#39

Post by notpachet »

bloopernova wrote: post #4

There's a shell script version of GNU parallel that's great for CI/CD pipeline tasks. You just keep it in your repo and source it as needed. It's incredibly useful, we use it in one build to batch process a few thousand things in groups of 25. Edited to add: finally got signed in to work, you create the script via: parallel --embed > scriptname.sh It's about 14,000 lines of awesome and works on "ash, bash, dash, ksh,…

Maybe this is a silly question, but what advantage do you get from checking that huge file into VC instead of just installing parallel ahead of time on the CI images?

Re: GNU Parallel, where have you been all my life?

#40

Post by wiredfool »

ssddanbrown wrote: post #14

Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick: find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}

Xargs is a nearly drop in replacement and probably already installed by default in most distros. You may need the -n 1 (one file per) and -P to parallelize.

  xargs -n 1 -P 8
Post reply on HN