Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel

gnu.org

Post reply on HN
Print view View original

61–70 of 76 posts

Re: GNU Parallel

#61

Post by paskozdilar »

kazinator wrote: post #51
paskozdilar wrote: post #23

Earlier quoted context omitted.

This is an excellent explanation of why "GNU finds this acceptable". Note that the citation message can also be easily silenced just by creating an empty file: touch ~/.parallel/will-cite

Tools shouldn't be probing the file system for files not related to the job they are doing, period. The right way to do this is to patch the behavior out of the program, which you're entitled to do by its license. Or, rather by the fact that the license doesn't concern itself with use. That is covered in the FAQ: == I do not like the notice. Can I fork GNU Parallel and remove it? == Yes. GNU Parallel is released unde…

> Tools shouldn't be probing the file system for files not related to the job they are doing, period.

You're entitled to your preferences and your own fork of the GNU Parallel. I'll even help you out with that - all you need to do to remove the citation message is to comment out lines 1840-1843 in "src/parallel" file.

Just please don't demand everyone else (i.e. distro maintainers) to stab the main developer in the back just to accommodate your preferences.

Re: GNU Parallel

#62

Post by MainJane »

musicale wrote: post #59
MainJane wrote: post #43

Earlier quoted context omitted.

> I don't care what GNU thinks, but it's simply not scalable. How so? A lot of software requires you to configure it before the first run, and we regard that as scalable. A lot of software requires you to pay for it before the first run (most Microsoft server software comes to mind), yet we regard that as scalable. You can also pay for gnu paralell: https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... Is i…

$ ls Thank you for using the /bin/ls utility! Did you know that you can upgrade to LS PRO for a mere fraction of a bitcoin? Or just post a selfie tagged #LS_PRO_RULES on Twitter! LS PRO has many amazing features that you are missing. This message can be removed by using the --no-awesome-ls-pro-upgrade-msg flag. Here is your file listing: .bashrc .catconf .cprc .ddconfig .dfprefs ... $ exit -1

Honestly, I fail to see the problem, if I had to run `ls --no-awesome-ls-pro-upgrade-msg` once when I installed it the first time. And if I did not like it, I could use one of the alternatives to `ls` or build my own.

In LibreOffice I have to click a "Don't show tip of the day again" every time I install it on an new machine, and personally I have no problem with that. If I had, I would use something else.

Zsh asks me to configure it, first time I run it. I find that slightly annoying, but not to the extend that I would even consider complaining, sending a patch, or using an alternative.

But I assume you are aware that your comparison is really not valid: Parellel is not limited in features - you do not get extra features by paying/citing. What you are doing is keeping it alive.

Also, if you really do not like the notice, why not just pay for it? Are you opposed to paying for free software? And if so, how do you suggest developers of free software make a living? And why are you not actively doing that for GNU Parallel, which you clearly have so strong opinions on, that you are willing to spend time complaining but not willing to ignore (and use another tool)?

Re: GNU Parallel

#63

Post by forgotmypw17 »

rurban wrote: post #60
MainJane wrote: post #47

Earlier quoted context omitted.

I was curious how much breakage GNU Parallel has suffered. So I fetched all versions (in parallel) and ran: parallel -k --tag --argsep -- {} echo ::: 1 -- parallel-* Every version since 20120622 work (except for 20121022). That is code which is almost 10 years old.

you need to try with all the perl versions, not the parallel versions.

In my anecdotal, n=1 experience, nothing Perl-based I've ever used has EVER broken over 20+ years, not even ONCE.

Compare this with PHP, whose breaking changes between releases has taken down my sites on multiple occasions.

Compare this with Python, whose breaking changes prevent me from running the overwhelming majority of Python things I've tried to use.

Re: GNU Parallel

#64

Post by desktopninja »

Can't quite pin my resistance to parallel but I've enjoyed using this magic for many moons now:

xargs -t -n 1 -P 3 -I TARGET_COMPUTER ssh -q TARGET_COMPUTER "

sleep \$(shuf -i 1-9 | head -1)

echo \$(hostname; date)

echo \"===> \$(hostname) > ~/I_WAS_HERE

" server-01

server-02

server-03

server-04

server-05 "

Re: GNU Parallel

#65

Post by cb321 »

MainJane wrote: post #57
cb321 wrote: post #54

Earlier quoted context omitted.

> allowing for the total output to be bigger than the free space on /tmp. I could not do it. This is an unreasonable standard when you do not know in advance how big the output is. What do you imagine GNU parallel does? Use `df` on every host it knows about to fill every disk partition it can? That sounds like a pretty system-hostile behavior to me. Meanwhile, putting your temp files somewhere bigger is obv. as easy…

> This is an unreasonable standard when you do not know in advance how big the output is. Why is that unreasonable? Let us say a single job outputs 10% of the free space. As long as you run fewer than 10 jobs in parallel, GNU paralel can run forever, because it spits out the output when a job is done and then frees up the space for this job, while starting the next one. A simple example: yes 1000000 | parallel -j10 s…

You are just moving goalposts from "grouping to not mix" in the comparison doc to "grouping to not mix with exact space management profile(s) of GNU parallel". Even worse, you now bring in IO space-speed assumptions, other use cases (hay generation not needle search), various dissembling and childish "taunts for proof" when you clearly understood the suggestion well enough to analyze it for potential limitations. Your attitude is the problem, not missing code. Also, I never said "/tmp" and the paths could be FIFOs with record size/buffering limitations instead.

Speaking of /tmp filling and questionable space management defaults:

    yes 2000000000 | parallel seq | pv > /dev/null
fills my /tmp disk partition (or $TMPDIR) before emitting one byte to pv with invisible (unlinked) temp files. Not ideal. GNU sort at least shows me there are files present yet also seems to clean up on Ctrl-C.

There is likely some solution to fix this in 15 kLOC of gross Perl. I did not find it in "5 minutes" (another unreasonable standard since the many 1000s of lines of GNU parallel docs take far longer to read, but you already seem to ignore my explanations of "unreasonable"). You even anticipate this in your 10% example. At least in my life, "way more" is often much more than 10x more. So, you basically contradict yourself.

As to the actual subtopic, besides being unfair/out-of-date, the comparison tableau is also incomplete - maybe willfully so, as per too common marketing dishonesty. "Proof?" People use parallelism to speed things up and need to make decisions about job granularity to not have perf killed by overhead. Some would say this matters more than 95% of the tableau evaluation points. Yet, no overhead benchmarks. Maybe they make GNU parallel look bad?

Re: GNU Parallel

#66

Post by cb321 »

With the default setup, GNU parallel seems to be about 1000x slower than it should be and 165x slower than serial xargs. On a 16 core/32 thread CPU running Linux 5.16 (with parallel-20220322, perl-5.34.1, ripgrep-13.0.0, findutils-4.9.0, grep-3.7):

    cd /dev/shm
    wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.2.tar.xz
    tar xpJf linux-5.17.2.tar.xz
    cd linux-5.17.2
    rg -a --no-ignore --files | tr \\n \\0 > ../f
    tm="/usr/bin/time"; lb="--line-buffered"
    $tm rg -a --no-ignore -l foofoo           # 130 milliseconds wall
    $tm xargs -0P16 grep $lb -l foofoo
GNU parallel uses massive %CPU to make slow progress which also tends to confuse users new to parallelism in general. Hey, maybe there is a --dont-go-slow flag somewhere. I'd bet actual new users take a while to find it - maybe they never find it. There seems to at least some story of bad/confusing defaults here. A quick grep in the parallel package for benchmark only seems to indicate things that benefit from network asynchrony, not CPU parallelism.

I tried to pick something anyone could reproduce, should they so desire. Cheers.

Re: GNU Parallel

#67

Post by cb321 »

cb321 wrote: post #66

With the default setup, GNU parallel seems to be about 1000x slower than it should be and 165x slower than serial xargs. On a 16 core/32 thread CPU running Linux 5.16 (with parallel-20220322, perl-5.34.1, ripgrep-13.0.0, findutils-4.9.0, grep-3.7): cd /dev/shm wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.2.tar.xz tar xpJf linux-5.17.2.tar.xz cd linux-5.17.2 rg -a --no-ignore --files | tr \\n \\0 > ../…

Addendum: I guess parallel is not a "drop-in" for xargs by default, but more like xargs -n1, as has been mentioned elsethread. You need to use -X (that is a capital 'x' as lowercase 'x' means something else following the xargs CLI syntax, but not its semantics, LOL [1]).

    $tm parallel -X0j16 grep -l foofoo
So, GNU parallel is "only" 20X slower than it should be and "only" 3X slower than serial xargs on a 16 core box (on bare metal, not in the cloud, by the way). Still pretty awful unless there is another "--dont-burn-down-forests-for-electricity" flag I'm missing.

[1] https://unix.stackexchange.com/questions/273170/gnu-parallel...

Re: GNU Parallel

#68

Post by caro11ne »

desktopninja wrote: post #64

Can't quite pin my resistance to parallel but I've enjoyed using this magic for many moons now: xargs -t -n 1 -P 3 -I TARGET_COMPUTER ssh -q TARGET_COMPUTER " sleep \$(shuf -i 1-9 | head -1) echo \$(hostname; date) echo \"===> \$(hostname) > ~/I_WAS_HERE " server-01 server-02 server-03 server-04 server-05 "

I use:

    iwashere() {
      sleep $(shuf -i 1-9 | head -1)
      date
      echo "===> $(hostname)  ~/I_WAS_HERE
    }
    env_parallel -Sserver-0{1..5} --tag --nonall iwashere
I like that I can try out the function locally before running it remotely.

I like that I do not have to give xargs a multiline argument.

I am terrible at quoting inside quoting inside quoting, so I like that I can simply avoid the quoting. I also think my colleagues will find it easier to read (thus maintain).

Re: GNU Parallel

#69

Post by caro11ne »

cb321 wrote: post #66

With the default setup, GNU parallel seems to be about 1000x slower than it should be and 165x slower than serial xargs. On a 16 core/32 thread CPU running Linux 5.16 (with parallel-20220322, perl-5.34.1, ripgrep-13.0.0, findutils-4.9.0, grep-3.7): cd /dev/shm wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.17.2.tar.xz tar xpJf linux-5.17.2.tar.xz cd linux-5.17.2 rg -a --no-ignore --files | tr \\n \\0 > ../…

I find this scary:

    $ export LC_ALL=C
    $ $tm xargs -0P1 grep $lb t
It greps for lines containing t, sorts the lines and computes a hash.

Note how "xargs -P16 grep" gives the wrong answer. The output from parallel matches exactly the lines from "xargs -P1". With "-k" the lines are even in the same order (sorting removed):

    $ $tm xargs -0P1 grep $lb t
I have not analyzed the output but I think the error is caused by the issue described here: https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_w...

How anyone would ever use "xargs -P16 grep" is beyond me. I honestly do not care how fast I can get an answer, if I cannot trust the answer is correct.

I can see someone claimed they could build a safe parallel grep, but seemed not to do so: https://news.ycombinator.com/item?id=30890780#30913304 It would have been interesting to see.

Re: GNU Parallel

#70

Post by MainJane »

cb321 wrote: post #65
MainJane wrote: post #57

Earlier quoted context omitted.

> This is an unreasonable standard when you do not know in advance how big the output is. Why is that unreasonable? Let us say a single job outputs 10% of the free space. As long as you run fewer than 10 jobs in parallel, GNU paralel can run forever, because it spits out the output when a job is done and then frees up the space for this job, while starting the next one. A simple example: yes 1000000 | parallel -j10 s…

You are just moving goalposts from "grouping to not mix" in the comparison doc to "grouping to not mix with exact space management profile(s) of GNU parallel". Even worse, you now bring in IO space-speed assumptions, other use cases (hay generation not needle search), various dissembling and childish "taunts for proof" when you clearly understood the suggestion well enough to analyze it for potential limitations. You…

If you feel I am "moving the goalposts" why not just prove your original case? If you are spending 5 minutes on reading the source code, why not instead spend them on proving your original assertion is correct? You can then let the readers decide if they feel I "move the goalposts".

I included the example:

    yes 1000000 | parallel -kj10 "echo 'This  is  double  spaced  '{#}; seq {}" | pv >/dev/null
to give you some fixed "goalposts" to aim for: Provide a solution that gives the same output byte for byte.

Also you do not seem to get the point about the amount of data. I regularly have output from a single job that is bigger than RAM, but rarely have output from a single job that would fill /tmp. However, the total combined output from all the jobs will often take up more space than /tmp.

In numbers: RAM=32 GB, /tmp=400 GB, a single job=33 GB, number of jobs=1000, jobs in parallel=8.

In other words: Running all jobs and saving the outputs into files before outputting data will not be useful for me. If you want to use FIFOs I really cannot see how you can deal with output that is bigger than RAM, unless you mix output from different jobs - which again would not be useful to me. But prove me wrong by spending 5 minutes on building the solution.

As for your example:

    yes 2000000000 | parallel seq | pv > /dev/null
How would you design this, if output from different jobs are not allowed to mix?

If they are allowed to mix paralel gives you:

    # bytes are allowed to mix
    yes 2000000000 | parallel -u seq | pv > /dev/null
    # only full lines are allowed to mix
    yes 2000000000 | parallel --lb seq | pv > /dev/null
none of these use space in /tmp.

I sit back with the feeling you are willing to spend hours complaining, but not 5 minutes on proving your assertion that it can be done "easy-ish".

Prove me wrong: Spend 5 minutes on the task you believed was "easy-ish".

If it cannot be done in 5 minutes, be brave enough to admit you were wrong.

Post reply on HN