Live data from Hacker News

GNU Parallel - build and execute command lines from standard input in parallel

savannah.gnu.org

21–30 of 37 posts

Re: GNU Parallel - build and execute command lines from standard input in parallel

#21
Ah, one of these again.

I wrote a simpler one a couple of years ago (http://code.google.com/p/spawntool/) myself. All it does is read commands from stdin, one per line, and keep a desired number of processes running until all command lines are exhausted. Simple.

I wrote my own because I got tired of all kinds of substitution and quoting issues with xargs. With spawn I only need to generate the shell commands and instead of piping them to bash I pipe them to spawn. Also, this means I can easily review your command line generation with less (so that quotes etc. are good) until I eventually switch to sh or spawn.

Re: GNU Parallel - build and execute command lines from standard input in parallel

#23
post #19
post #3

This is very cool, but a little opaque at first read... To get a quicker and digestible intro, watch the video introduction (linked on the page): http://www.youtube.com/watch?v=OpaiGYxkSuQ

Well, the example(at least the first one) in that video is bit skewed. First, he runs gzip and then immediately runs 'parallel gzip' without dropping disk caches. So in the later case the bottleneck would be CPU rather than disk IO(everything read from disk cache in the RAM). IMO I expect for the work that is IO bound we won't see any significant improvement using parallel or anything similar.

Ideas for next video are most welcome. The ideal task:

1. Is single threaded 2. Takes a lot of CPU 3. Is a task that everyone can understand and relate to and which is close to a real world scenario

I have loads of examples meeting requirement 1+2. It is 3 that is the hard part.

Post them to parallel@gnu.org

Re: GNU Parallel - build and execute command lines from standard input in parallel

#28
post #6

It seems like 90% of the uses for this can be taken care of with xargs: echo "file1 file2" | xargs -P 2 gzip

As I understand it, xargs only runs on the local machine; GNU parallel can run on remote machines as well. So parallel is the cluster-friendly version of xargs's -P.

Yep. There also is dxargs which looks very useful http://www.semicomplete.com/blog/geekery/distributed-xargs.h...

Re: GNU Parallel - build and execute command lines from standard input in parallel

#29

Someone managed to get this working on os x? Standard "./configure && make && make install" outputs errors.

The MacPorts version worked for me.

Perhaps the Portfile will have the patches you need to manually compile on OS X:

http://trac.macports.org/browser/trunk/dports/sysutils/paral...

Re: GNU Parallel - build and execute command lines from standard input in parallel

#30
post #17

I'll express here what I feel while reading any man pages.. Examples should be at the top! (10 years of frustration in that one line message :p)

I've just resigned to do a search for /EXAMPLES as soon as I need any.
Post reply on HN