Further overloading this overloaded utility name screws up packagers and hinders usage of any of the variants. Rename it please.
Parallel – A command-line CPU load balancer written in Rust
21–30 of 65 posts
Re: Parallel – A command-line CPU load balancer written in Rust
#22Why this can be faster than GNU parallel?
GNU Parallel is extremely sluggish because it does all sort of different things behind your back: It buffers output on disk (so the from different jobs are not mixed and you are not limited by the amount of RAM - it will even compress the buffering if you are short on disk space), it checks if the disk is full for every job (so you do not end up with missing output), it gives every process its own process group (so the process with children can be killed reliably with --timeout and --memfree), and a lot of other stuff.
It lets you code your own replacement string (using --rpl), and lets you make composed commands with shell syntax:
myfunc() { echo joe $*; }
export -f myfunc
parallel 'if [ "{}" == "a" ] ; then myfunc {} > {}; fi' ::: a b c
yes `seq 10000` | parallel true
You can read a lot more about the design in `man parallel_design` and see the evolution of overhead time per job compared to each release on: https://www.gnu.org/software/parallel/process-time-j2-1700MH...
In other words: Treat GNU Parallel as the reliable Volvo that has a lot of flexibility and will get the job done with no nasty corner case surprises.
It is no doubt possible to make a better specialized tool for situations where the overhead of a few ms per job is an issue and where you neither need brakes, seatbelts nor airbags. xargs is an example of such a tool, and you can have both GNU Parallel and xargs installed side by side.
Re: Parallel – A command-line CPU load balancer written in Rust
#23Earlier quoted context omitted.
He lost me at the point when he complained that GNU parallel "includes the ability to transfer files between computers". For me at least that is _the_ feature of GNU parallel that actually makes it really useful. Which I guess is the problem with all these discussions, one persons useless bloat is another persons nr. 1 killer feature.
But in the spirit of Unix, shouldn't parallel exec some file transfer program?
It does: Rsync.
Re: Parallel – A command-line CPU load balancer written in Rust
#24Earlier quoted context omitted.
He lost me at the point when he complained that GNU parallel "includes the ability to transfer files between computers". For me at least that is _the_ feature of GNU parallel that actually makes it really useful. Which I guess is the problem with all these discussions, one persons useless bloat is another persons nr. 1 killer feature.
But in the spirit of Unix, shouldn't parallel exec some file transfer program?
The actual Perl code calls out to ssh and rsync (or can be configured to use something else) when it's time to actually connect and transfer files. It just does it in a way that is nice and reasonably transparent to the end user.
It felt like his complaint was that that was 'bloat' since Real Men can achieve almost the same thing by just piping some output through some bash scripts they just hacked together.
Re: Parallel – A command-line CPU load balancer written in Rust
#25Earlier quoted context omitted.
My point wasn't optimization. Just that the author was making it more complicated than was needed. It's like discovering that "ls" is spawning 5 threads for internal communication.
In a language with good threading support, internal threads are typically making things easier rather than harder. I tend to spawn lots of threads in Rust just because I can and it simplifies the code a lot over having to do some async callback mess. In particular there is no sane way to async waitpid() on POSIX.
pselect / self-pipe / etc. on SIGCHLD, and on receipt of SIGCHLD, loop through all known child processes with wait(WNOHANG).
... I'm not sure I can disagree with "no sane way".
Re: Parallel – A command-line CPU load balancer written in Rust
#26Earlier quoted context omitted.
a version in C that I think was first released before GNU parallel is in "moreutils" https://joeyh.name/code/moreutils/ A few years ago, debian made GNU parallel provide the "/usr/bin/parallel" executable, instead of moreutils. The maintainer of moreutils had some interesting things to say about that: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050#75
> It is 5143 lines of code, and the anthethesis of a simple, clean, well-designed unix command. Only 5k? :D But, yes, the criticisms are valid. I recommend moreutils.
One of the reasons for the high line number count is the decision not to depend on modules that are not part of the core of Perl 5.8.
Quite a few of the lines are to deal with different flavours of operating systems.
The benefit of this is that by copying the same single file you can have GNU Parallel running on FreeBSD 8, Centos 3.9, and Cygwin.
Re: Parallel – A command-line CPU load balancer written in Rust
#27Earlier quoted context omitted.
But in the spirit of Unix, shouldn't parallel exec some file transfer program?
The actual Perl code calls out to ssh and rsync (or can be configured to use something else) when it's time to actually connect and transfer files. It just does it in a way that is nice and reasonably transparent to the end user. It felt like his complaint was that that was 'bloat' since Real Men can achieve almost the same thing by just piping some output through some bash scripts they just hacked together.
And it is exactly the hacking part that GNU Parallel tries to help with: A lot of the helper functions in GNU Parallel could be done by expert users (--nice, --tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer, --return, --cleanup, --(n)onall).
But non-expert users will invariably make mistakes (e.g. get quoting wrong, not getting remote jobs to die if the controlling process is killed, or re-scheduling jobs that were killed by --timeout), and why not just have small wrapper scripts built into GNU Parallel that are well-tested, so the non-expert users can enjoy the same stability as the expert users?
Re: Parallel – A command-line CPU load balancer written in Rust
#28Earlier quoted context omitted.
The actual Perl code calls out to ssh and rsync (or can be configured to use something else) when it's time to actually connect and transfer files. It just does it in a way that is nice and reasonably transparent to the end user. It felt like his complaint was that that was 'bloat' since Real Men can achieve almost the same thing by just piping some output through some bash scripts they just hacked together.
And it is exactly the hacking part that GNU Parallel tries to help with: A lot of the helper functions in GNU Parallel could be done by expert users (--nice, --tmux, --pipepart, env_parallel, --compress, --fifo, --cat, --transfer, --return, --cleanup, --(n)onall). But non-expert users will invariably make mistakes (e.g. get quoting wrong, not getting remote jobs to die if the controlling process is killed, or re-sche…
Having written my fair share of those hacky wrapper scripts before I discovered GUN parallel I certainly am very happy that they offer everything I need in a single easy to use command.
Re: Parallel – A command-line CPU load balancer written in Rust
#29I always tended to autogenerate a makefile with fake goals like:
g1:
job1
g2:
job2
Re: Parallel – A command-line CPU load balancer written in Rust
#30Earlier quoted context omitted.
One possible reason is that GNU parallel is a perl script.
a version in C that I think was first released before GNU parallel is in "moreutils" https://joeyh.name/code/moreutils/ A few years ago, debian made GNU parallel provide the "/usr/bin/parallel" executable, instead of moreutils. The maintainer of moreutils had some interesting things to say about that: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050#75
> a version in C that I think was first released before GNU parallel
While this is technically correct, it is misleading: GNU Parallel existed before it became GNU. See details on: https://www.gnu.org/software/parallel/history.html