Live data from Hacker News

GNU Parallel

gnu.org

11–20 of 76 posts

Re: GNU Parallel

#11
post #2

It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance…

Your concern about this being non-free is addressed by an entire FAQ. Here's a link to the FAQ page from the software's Git repo:

https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...

Re: GNU Parallel

#12

This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.... It has a lot of features that can feel excessive at first glance but if you have felt some pain in building jobs, most of it is pretty sensible and much better than rollyourown.

The comparison is not very fair to modern day xargs.

`nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism.

Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`.

Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh` targets, possibly duplicates if you want to run >1 job per host. (In pure POSIX sh you could use eval and $1, $2 positional args with shell arithmetic..)

Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.

Re: GNU Parallel

#13
post #2

It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance…

[deleted]

Re: GNU Parallel

#14
post #5
post #2

It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance…

I don't understand how GNU finds this acceptable...

[edited for formatting and to fix Git link]

According to https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...

> == Is the citation notice compatible with GPLv3? ==

> Yes. The wording has been cleared by Richard M. Stallman to be compatible with GPLv3. This is because the citation notice is not part of the license, but part of academic tradition.

> Therefore the notice is not adding a term that would require citation as mentioned on: https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation

If you are of the view that clearance by RMS is clearance by FSF/GNU, then that's how they find it acceptable. If you take a different view, then the next part of that section applies:

> If you disagree with Richard M. Stallman's interpretation and feel the citation notice does not adhere to GPLv3, you should treat the software as if it is not available under GPLv3. And since GPLv3 is the only thing that would give you the right to change it, you would not be allowed to change the software.

There's also an interesting comparison to be made:

> == How do I silence the citation notice? ==

> Run this once:

> parallel --citation

> It takes less than 10 seconds to do and is thus comparable to an 'OK. Do not show this again'-dialog box seen in LibreOffice, Firefox and similar programs.

Re: GNU Parallel

#15
post #10
post #2

It's nice, but the citation bit strikes me as very non-free: > If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance…

It doesn't affect your right to do anything with the software. It doesn't even affect your right to remove this citation notice (and some distributions do so). Free software is not about software not being annoying; merely about your right to remove the annoyances. I'm not certain GNU parallel's approach to obtaining funding is a good strategy, but I find it weird that people object to it on philosophical or legal gr…

> I'm not certain GNU parallel's approach to obtaining funding is a good strategy,

Indeed, it's a good question! I know that, for the SGI UV300 we got through an NIH S10 grant, the usage of the supercomputer is tracked by looking for the grant number in publication acknowledgements. Yes, we already got the money, so you may wonder "what does it matter?", but our ability to get future funding (especially as the UV300 nears retirement age) depends (at least in part) on showing how well we used our previous funding.

In our case, the funding came through an NIH grant, so we ask people reference the grant number. But more broadly, and especially for software, issuing a DOI for released versions (through a service like Zenodo https://zenodo.org), along with a request for acknowledgement, gives a way to track usage. For example, the 20150322 release of parallel (DOI 10.5281/zenodo.16303) has been cited at least 63 times (per https://doi.org/10.5281/zenodo.16303).

Looking at https://scholar.google.com/citations?user=D7I0K34AAAAJ&hl=en, it seems the 2011 version of Parallel has been cited over 1,000 times.

The author is at the University of Copenhagen (per the Google Scholar link above), so it's entirely possible that at least some of the funding for his employment is coming from sources that use citation counts as an indicator that they are "getting their money's worth" by continuing to fund (at least part of) Mr. Tange's employment.

Re: GNU Parallel

#16
post #5

Earlier quoted context omitted.

I don't understand how GNU finds this acceptable...

[edited for formatting and to fix Git link] According to https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... > == Is the citation notice compatible with GPLv3? == > Yes. The wording has been cleared by Richard M. Stallman to be compatible with GPLv3. This is because the citation notice is not part of the license, but part of academic tradition. > Therefore the notice is not adding a term that would requir…

[deleted]

Re: GNU Parallel

#17
post #12

This can be a good entry into the "why" for many people that use xargs now - https://www.gnu.org/software/parallel/parallel_alternatives.... It has a lot of features that can feel excessive at first glance but if you have felt some pain in building jobs, most of it is pretty sensible and much better than rollyourown.

The comparison is not very fair to modern day xargs. `nproc` is a relatively standard utility (coreutils). So, xargs -P$(nproc) gets you core (or core-proportional) parallelism. Grouping output/Making a safe parallel grep is also easy-ish with `--process-slot-var=slot` and sending to `tmpOut.$slot`. Jobs on remote computers can be done similarly with any kind of `arrayVar[$slot]` setup where `arrayVar` has a bunch of…

> Anyway, those three are just off the top of my head, unfairness-wise. Last I looked at the source for GNU parallel it looked like mountains upon mountains of Perl I would rather not depend upon, personally, but to each his own.

Well, there was a Rust version with zero Perl, now unfortunately archived. It wasn't 100% on a par with the original and wasn't really finished. On the other hand, built easily for Windows and helped me on a few occasions.

https://github.com/mmstick/parallel

Re: GNU Parallel

#18
parallel is so useful and i use it multiple times daily. i wish its `:::` syntax was supported at the shell level so i could use it for every application

Re: GNU Parallel

#19
GNU Parallel is one of my favorite utilities of all time.

I used to write complex and bug-ridden scripts that used bash job control to implement parallel execution of batch jobs. When I discovered GNU Parallel, I deleted them all and never looked back.

Also, the documentation is awesome - there is a whole book [0] on GNU Parallel, and the manpage even links to a series of youtube videos [1] that explain how it works.

[0] https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pd...

[1] http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Re: GNU Parallel

#20
If you want to execute commands on multiple remote hosts over SSH, just run a tmux session and launch a tab on each host in a loop and execute them.

It's far easier to follow the output and individually deal with prompts.

Post reply on HN