Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel

gnu.org

Post reply on HN
Print view View original

51–60 of 76 posts

Re: GNU Parallel

#51

Post by kazinator »

paskozdilar wrote: post #23
CaliforniaKarl wrote: post #14

Earlier quoted context omitted.

[edited for formatting and to fix Git link] According to https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... > == Is the citation notice compatible with GPLv3? == > Yes. The wording has been cleared by Richard M. Stallman to be compatible with GPLv3. This is because the citation notice is not part of the license, but part of academic tradition. > Therefore the notice is not adding a term that would requir…

This is an excellent explanation of why "GNU finds this acceptable". Note that the citation message can also be easily silenced just by creating an empty file: touch ~/.parallel/will-cite

Tools shouldn't be probing the file system for files not related to the job they are doing, period.

The right way to do this is to patch the behavior out of the program, which you're entitled to do by its license. Or, rather by the fact that the license doesn't concern itself with use.

That is covered in the FAQ:

  == I do not like the notice. Can I fork GNU Parallel and remove it? ==

  Yes. GNU Parallel is released under GNU GPLv3 and thus you are allowed
  to fork the code. But you have to make sure that your forked version
  cannot be confused with the original, so for one thing you cannot call
  it anything similar to GNU Parallel as that would cause confusion
  between your forked version and the original.
If you're not redistributing it, this doesn't apply to you; you're only using the program.

I believe that a distro could get around this by providing a script a user can execute, or a patch that the user can apply that removes the nag code from the installation of GNU Parallel. Even if we take the view that the script creates a fork, it only creates a private one on the user's machine, and not anything that is redistributed; without distribution taking place, what is taking place is use.

Re: GNU Parallel

#52

Post by cb321 »

arrakeen wrote: post #34
cb321 wrote: post #12

Earlier quoted context omitted.

The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of…

> Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own. i used parallel for years under the assumption that it was written in C and only recently learned it was written in perl when i decided to dive deeply into its documentation. if you're using a package manager to install parallel and it runs fast enough for you…

Some users like to add their own features or have problems to debug. They surely care. Others may want to move coordination to a remote host and care about some single-file transfer of the exact version of the tool. They also care. It's ok that you don't, personally, of course.

Re: GNU Parallel

#53

Post by cb321 »

arrakeen wrote: post #35
cb321 wrote: post #22

Earlier quoted context omitted.

Well, some archived project is not so great either. The core functionality is not even a 20 line bash script since bash grew wait -n, though: #!/bin/bash if [ "${1-0}" -lt 1 ]; then # No arg / arg not a number >= 1 echo "Usage: $0 "; echo "reads cmds from stdin, running up to N at once." exit 1 fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1 while read cmd; do jobs > $TMP #…

sure a "parallel xargs" can ostensibly be implemented in POSIX sh but that's merely the tip of the iceberg with what parallel can do. why not just skim the documentation and give it a try?

I have skimmed. I did give it a try. Not for me.

Re: GNU Parallel

#54

Post by cb321 »

MainJane wrote: post #50
cb321 wrote: post #12

Earlier quoted context omitted.

The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of…

> The comparison is not very fair to modern day xargs. I am curious how you come to that conclusion. > `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. I follow you on this point. A bit harder on remote systems, but definitely doable. > Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to…

> allowing for the total output to be bigger than the free space on /tmp. I could not do it.

This is an unreasonable standard when you do not know in advance how big the output is. What do you imagine GNU parallel does? Use `df` on every host it knows about to fill every disk partition it can? That sounds like a pretty system-hostile behavior to me.

Meanwhile, putting your temp files somewhere bigger is obv. as easy as $TMPDIR or such.

Best wishes/luck. I only have 5 minutes to explain why nothing can do the impossible like read a user's mind about disk free space management or the value of partial results. All software makes some assumptions... :-)

Re: GNU Parallel

#55

Post by cb321 »

smegsicle wrote: post #37
cb321 wrote: post #12

Earlier quoted context omitted.

The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of…

> mountains upon mountains of Perl I would rather not depend upon how bad was the perl?

Subjective. Judge for yourself: https://git.savannah.gnu.org/git/parallel.git

Re: GNU Parallel

#56

Post by arunix »

rurban wrote: post #44
forgotmypw17 wrote: post #26

Earlier quoted context omitted.

Perl has retained stable backwards compatibility and no breaking changes for 20+ years. What's wrong with Perl?

Lots of unstable breakages over the last year's. automake got broken by an unnecessary deprecation, signatures got broken, encodings, and dozens more. but still miles better than other such languages, and esp. if it would have been written in C, just as the incompatible moreutils counterpart.

> signatures got broken

Subroutine signatures are an experimental feature in Perl. Or are you referring to something else?

Re: GNU Parallel

#57

Post by MainJane »

cb321 wrote: post #54
MainJane wrote: post #50

Earlier quoted context omitted.

> The comparison is not very fair to modern day xargs. I am curious how you come to that conclusion. > `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. I follow you on this point. A bit harder on remote systems, but definitely doable. > Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to…

> allowing for the total output to be bigger than the free space on /tmp. I could not do it. This is an unreasonable standard when you do not know in advance how big the output is. What do you imagine GNU parallel does? Use `df` on every host it knows about to fill every disk partition it can? That sounds like a pretty system-hostile behavior to me. Meanwhile, putting your temp files somewhere bigger is obv. as easy…

> This is an unreasonable standard when you do not know in advance how big the output is.

Why is that unreasonable?

Let us say a single job outputs 10% of the free space. As long as you run fewer than 10 jobs in parallel, GNU paralel can run forever, because it spits out the output when a job is done and then frees up the space for this job, while starting the next one.

A simple example:

    yes 1000000 | parallel -j10 seq | pv >/dev/null
On my laptop I get 600 MB/s which would fill /tmp in a few minutes, and it does not.

When dealing with big data it is not uncommon that the total data piped between commands is way larger than the free space on /tmp (which is typically fast, where as free space on $HOME is slow - thus setting $TMPDIR to $HOME/tmp may slow down your job drastically).

If you only have 5 minutes, I hope you will use them on providing actual code to support your claim, that "The comparison is not very fair to modern day xargs."

If it takes longer than 5 minutes to code, I would say your use of "easy-ish" is unwarranted.

You leave me with the feeling that you have not thought this through and that the reason why you do not provide any code is because you are now realizing you are wrong, but you do not have the guts to admit so.

Prove me wrong by posting the code. It should be "easy-ish" :)

You can use this as the test case to implement:

    yes 1000000 | parallel -kj10 "echo 'This  is  double  spaced  '{#}; seq {}" | pv >/dev/null

Re: GNU Parallel

#58

Post by [deleted] »

MainJane wrote: post #43
musicale wrote: post #28

Earlier quoted context omitted.

I don't care what GNU thinks, but it's simply not scalable. Imagine a world where every utility has its own irritating nag message that needs to be turned off.

> I don't care what GNU thinks, but it's simply not scalable. How so? A lot of software requires you to configure it before the first run, and we regard that as scalable. A lot of software requires you to pay for it before the first run (most Microsoft server software comes to mind), yet we regard that as scalable. You can also pay for gnu paralell: https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... Is i…

[deleted]

Re: GNU Parallel

#59

Post by musicale »

MainJane wrote: post #43
musicale wrote: post #28

Earlier quoted context omitted.

I don't care what GNU thinks, but it's simply not scalable. Imagine a world where every utility has its own irritating nag message that needs to be turned off.

> I don't care what GNU thinks, but it's simply not scalable. How so? A lot of software requires you to configure it before the first run, and we regard that as scalable. A lot of software requires you to pay for it before the first run (most Microsoft server software comes to mind), yet we regard that as scalable. You can also pay for gnu paralell: https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... Is i…

    $ ls
    Thank you for using the /bin/ls utility!
    Did you know that you can upgrade to LS PRO for a mere fraction of a bitcoin? 
    Or just post a selfie tagged #LS_PRO_RULES on Twitter!
    LS PRO has many amazing features that you are missing.
    This message can be removed by using the --no-awesome-ls-pro-upgrade-msg flag.
    Here is your file listing:
    .bashrc .catconf .cprc .ddconfig .dfprefs  ...
    $ exit -1

Re: GNU Parallel

#60

Post by rurban »

MainJane wrote: post #47
rurban wrote: post #44

Earlier quoted context omitted.

Lots of unstable breakages over the last year's. automake got broken by an unnecessary deprecation, signatures got broken, encodings, and dozens more. but still miles better than other such languages, and esp. if it would have been written in C, just as the incompatible moreutils counterpart.

I was curious how much breakage GNU Parallel has suffered. So I fetched all versions (in parallel) and ran: parallel -k --tag --argsep -- {} echo ::: 1 -- parallel-* Every version since 20120622 work (except for 20121022). That is code which is almost 10 years old.

you need to try with all the perl versions, not the parallel versions.
Post reply on HN