Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel – A command-line CPU load balancer written in Rust

github.com

Post reply on HN
Print view View original

11–20 of 65 posts

Re: Parallel – A command-line CPU load balancer written in Rust

#11

Post by bmalehorn »

The code is more complicated than it needs to be. It spawns N threads, then each thread forkexecs child processes. These threads communicate through an atomic int.

There is no need for threads. Just spawn background processes:

    echo 1 &
    echo 2 &
    wait
    echo 3 &
    wait
    echo 4 &
    ...
The key is that the wait() system call will hang until any child process finishes.

Re: Parallel – A command-line CPU load balancer written in Rust

#12

Post by the_mitsuhiko »

bmalehorn wrote: post #11

The code is more complicated than it needs to be. It spawns N threads, then each thread forkexecs child processes. These threads communicate through an atomic int. There is no need for threads. Just spawn background processes: echo 1 & echo 2 & wait echo 3 & wait echo 4 & ... The key is that the wait() system call will hang until any child process finishes.

I doubt that will make it measurable more efficient. Threads are cheap.

Re: Parallel – A command-line CPU load balancer written in Rust

#13

Post by aangjie »

CJefferson wrote: post #10
easytiger wrote: post #8

Earlier quoted context omitted.

Err, no. It execs other processes. the runtime overhead of running the interpreter is irrelevant as it provides no overhead over the general runtime of the sub-process.

GNU Parallel takes a surprising amount of CPU time. It does have various tasks (track it's children, feed them input, gather all their output and output it to the screen in the correct order), but I'm still surprised how much CPU it takes.

Hmm.. The example put up on the github README is a bad test case(for performance comparison), as it does almost no processing on the actual subprocesses. A better one could be creating thousands of small files(with dd or something in a directory, though it's io-bound not cpu-bound). Best might be some kind of repeated floating point exponentiation repeatedly.

Re: Parallel – A command-line CPU load balancer written in Rust

#14

Post by bmalehorn »

the_mitsuhiko wrote: post #12
bmalehorn wrote: post #11

The code is more complicated than it needs to be. It spawns N threads, then each thread forkexecs child processes. These threads communicate through an atomic int. There is no need for threads. Just spawn background processes: echo 1 & echo 2 & wait echo 3 & wait echo 4 & ... The key is that the wait() system call will hang until any child process finishes.

I doubt that will make it measurable more efficient. Threads are cheap.

My point wasn't optimization. Just that the author was making it more complicated than was needed.

It's like discovering that "ls" is spawning 5 threads for internal communication.

Re: Parallel – A command-line CPU load balancer written in Rust

#16

Post by kwoff »

Like headlines that are questions (answer: "no"; next), I tend to skip over headlines that say "written in $language". Usually means they've copied something that already exists, or it doesn't have much else distinctive about it than being written in that language.

Re: Parallel – A command-line CPU load balancer written in Rust

#17

Post by dagw »

paulddraper wrote: post #9
ploxiln wrote: post #6

Earlier quoted context omitted.

a version in C that I think was first released before GNU parallel is in "moreutils" https://joeyh.name/code/moreutils/ A few years ago, debian made GNU parallel provide the "/usr/bin/parallel" executable, instead of moreutils. The maintainer of moreutils had some interesting things to say about that: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=597050#75

> It is 5143 lines of code, and the anthethesis of a simple, clean, well-designed unix command. Only 5k? :D But, yes, the criticisms are valid. I recommend moreutils.

He lost me at the point when he complained that GNU parallel "includes the ability to transfer files between computers". For me at least that is _the_ feature of GNU parallel that actually makes it really useful. Which I guess is the problem with all these discussions, one persons useless bloat is another persons nr. 1 killer feature.

Re: Parallel – A command-line CPU load balancer written in Rust

#18

Post by the_mitsuhiko »

bmalehorn wrote: post #14
the_mitsuhiko wrote: post #12

Earlier quoted context omitted.

I doubt that will make it measurable more efficient. Threads are cheap.

My point wasn't optimization. Just that the author was making it more complicated than was needed. It's like discovering that "ls" is spawning 5 threads for internal communication.

In a language with good threading support, internal threads are typically making things easier rather than harder. I tend to spawn lots of threads in Rust just because I can and it simplifies the code a lot over having to do some async callback mess.

In particular there is no sane way to async waitpid() on POSIX.

Re: Parallel – A command-line CPU load balancer written in Rust

#19

Post by oblio »

dagw wrote: post #17
paulddraper wrote: post #9

Earlier quoted context omitted.

> It is 5143 lines of code, and the anthethesis of a simple, clean, well-designed unix command. Only 5k? :D But, yes, the criticisms are valid. I recommend moreutils.

He lost me at the point when he complained that GNU parallel "includes the ability to transfer files between computers". For me at least that is _the_ feature of GNU parallel that actually makes it really useful. Which I guess is the problem with all these discussions, one persons useless bloat is another persons nr. 1 killer feature.

But in the spirit of Unix, shouldn't parallel exec some file transfer program?
Post reply on HN