Viewing profile — ole_tange

Examples simulated with GNU Parallel: simple_gpu_scheduler --gpus 0 1 2 5 and skip() =}' python3 train_dnn.py --lr {lr} --batch_size {bs} ::: lr 0.001 0.0005 0.0001 ::: bs 32 64 12…

Fun fact: GNU Parallel used to be a wrapper for making Makefiles: https://www.gnu.org/software/parallel/history.html Also see this if you have a system without GNU Parallel: http:/…

--line-buffer may or may not give additional speed up.

--pipe is well know for being slow. Try --pipe-part instead: parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk | awk -f reduce.awk

> Granted, it's always possible that the parallel on $PATH might not be the one you're thinking of, but... Whatevs. For exactly this reason, GNU Parallel has the option --minversio…

> Same syntax applies. Except that is not entirely true: For instance according to the author the '{= perl expression =}' will probably never be supported. (Full disclosure: I am t…

Rust-parallel _is_ fast, and there is clearly a niche here, that GNU Parallel is unlikely to fill: By design GNU Parallel will never require a compiler; this is so you can use GNU …

The page with differences to other tools will be moved to 'man parallel_aternatives' in next version. It made sense when the section was short, but it has grown so big that it can …

Except for GNU Parallel --pipe. When you use '<' what you typically mean is '--pipepart -a file'.

Sure do. Makes you feel that the future is now!

Did you start by reading the "Reader's guide" which is presented before the first option is introduced in the man page?

Do you have an example of "piping next to process invocation"? It sounds interesting.

Just be aware that Stderr will contain stuff you did not ask for: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B... The syntax difference is small for simple tasks.

A good start is looking at the section 'Remote execution' in 'man parallel_tutorial'. Spoiler: You can have GNU Parallel kick the machines if the fail, and rerun the job on another…

The options section is no doubt for people, who know the tool. But the huge section with examples is not. Did you read the "Reader's guide" which is presented before the first opti…

A lot of effort has gone into the documentation of GNU Parallel: Intro videos, a one hour tutorial, tons of examples, a man page on the design decisions behind the code, and even a…

Can I ask to to run these and then comment? LESS=+/EXAMPLE: man parallel LESS=+/Reader.s man parallel man parallel_tutorial

Fun fact: GNU Parallel was originally a wrapper script that used `make` for the parallelization: https://www.gnu.org/software/parallel/history.html

"inside" = temporary files - not RAM. See comparison on https://www.gnu.org/software/parallel/parallel_design.html#B...

I hope you wont take this the wrong way, but can you elaborate on why you feel Tollef's parallel (from moreutils) is better than GNU Parallel?

Using Makefiles, however, depends on the results being files. If that is not the case, make cannot tell how far you got. GNU Parallel has --joblog and can continue from where it le…

Bigger list: https://www.gnu.org/software/parallel/man.html#DIFFERENCES-B...

From: https://www.gnu.org/software/parallel/history.html > parallel dates back to around the same time. It was originally a wrapper that generated a makefile and used make -j to do…

Hitachi already made your theme song: https://www.youtube.com/watch?v=xb_PyKuI7II :)