Ever read the man page for xargs? It reads just like the perscribing information for VIOXX.
Using xargs to do Parallel Processing
11–17 of 17 posts
Re: Using xargs to do Parallel Processing
#12make
Re: Using xargs to do Parallel Processing
#13Earlier quoted context omitted.
I tried it out on my late 2008 MacBook Pro (Lion, CoreDuo, SSD). Doesn't look so hot: Eds-MacBook-Pro workspace$ time (find . -name '*java'|parallel grep Message > /dev/null) real 0m2.176s user 0m1.111s sys 0m1.789s Eds-MacBook-Pro workspace$ time (find . -name '*java'|xargs grep Message > /dev/null) real 0m0.097s user 0m0.035s sys 0m0.061s GNU Parallel was installed using homebrew. I don't know what compiler flags w…
I didn't realise that parallel is a Perl script, which means it's going to lose when running very short-lived processes, as in my example.
True. Unless you need some of the facilities in GNU Parallel that you do not find in xargs (such as --group which is default, running on remote machines, -L with -I, or --pipe for processing piped data).
Re: Using xargs to do Parallel Processing
#14Meet GNU Parallel : http://www.gnu.org/software/parallel/
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Re: Using xargs to do Parallel Processing
#15My most common use of xarsg -P is for compressing logs on remote hosts before downloading them: cat RemoteHosts | xargs -P 0 -n 1 -I Host rsh Host gzip -9 xxx/*.log The cool thing is that it waits until the completion of all the commands before returning.
With GNU Parallel it looks like this:
parallel --nonall --slf RemoteHosts -j0 gzip -9 xxx/*.log
parallel --nonall --slf RemoteHosts -j0 --arg-sep , parallel gzip -9 ::: xxx/*.logRe: Using xargs to do Parallel Processing
#16If you are compressing a single large file, you may find "pigz" of interest -- it's a gzip that uses multiple threads pretty well.
Re: Using xargs to do Parallel Processing
#17My most common use of xarsg -P is for compressing logs on remote hosts before downloading them: cat RemoteHosts | xargs -P 0 -n 1 -I Host rsh Host gzip -9 xxx/*.log The cool thing is that it waits until the completion of all the commands before returning.
rsh or ssh? All my VPSs have rsh disabled.