Live data from Hacker News

GNU Parallel Cheat Sheet [pdf]

gnu.org

41–50 of 64 posts

Re: GNU Parallel Cheat Sheet [pdf]

#41
Each time I've seen something about GNU parallel pop up I've been tempted to post, but I've never made an account until now.

I wrote a very different style of command parallelizer that I named lateral. It doesn't require constructing elaborate commandlines that define all of your work at once. You start a server, and separate invocations of 'lateral run' add your commands to a queue to run on the server, including their filedescriptors. It makes for easier parallelization of complex arguments.

Take a look if this sort of thing interests you, as I haven't seen anyone write one like this before. Its primary difference is the ease with which each separate command can output to its own log, and the lack of need to play games with shell quoting and positional arguments.

Check it out: https://github.com/akramer/lateral

Re: GNU Parallel Cheat Sheet [pdf]

#44

Earlier quoted context omitted.

The multiprocessing module is pretty good in Python.

Logging was lost last time I tried it

Has this issue recently. Turns out there's a great library for this specifically. https://github.com/jruere/multiprocessing-logging

Re: GNU Parallel Cheat Sheet [pdf]

#45
post #3

I've never used GNU Parallel. But could someone explain to me the value add vs GNU xargs -P/--max-procs? From the examples at the top, it seems like those could be achieved with xargs.

A couple months ago, I parallelized execution of thousands of slow batch jobs on a fleet of remote servers. With parallel that was one command, including estimated time to completion and retries for failed jobs. It was awfully nice not to need to install or setup anything or spend time coding built-in features. Once it was done, I will almost certainly never run that exact operation again.

I normally use xargs for simple things and if it’s a regular business operation I’d setup a task queue but there’s a fair amount of work in the middle where it’s nice to have a solid tool with most of the features you could want built in and tested.

Re: GNU Parallel Cheat Sheet [pdf]

#46

Ah, the rare case of nagware in GNU. From the man page: "--citation Print the BibTeX entry for GNU parallel and silence citation notice. If it is impossible for you to run --bibtex you can use --will-cite. If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do no…

Beyond being supremely irritating, nagware is simply not scalable.

Imagine if every utility, library, or driver in a typical Linux distribution took this approach. :(

I encourage Debian et al. to adopt a "no nagware" policy.

Re: GNU Parallel Cheat Sheet [pdf]

#47

Ah, the rare case of nagware in GNU. From the man page: "--citation Print the BibTeX entry for GNU parallel and silence citation notice. If it is impossible for you to run --bibtex you can use --will-cite. If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do no…

Do you feel this is covered in the FAQ? http://git.savannah.gnu.org/cgit/parallel.git/plain/doc/cita...

The FAQ makes it worse:

"GNU Parallel is indirectly funded through citations. It is therefore important for the long term survival of GNU Parallel that it is cited. The citation notice makes users aware of this."

It's a bit like saying:

"Webkit is indirectly funded by iPhones. It is therefore important for the long term survival of Webkit that people purchase iPhones. The iPhone notice make users aware of this."

Re: GNU Parallel Cheat Sheet [pdf]

#48
post #40

Earlier quoted context omitted.

I enjoy following your thought process. I cannot make that fit with the content of the FAQ: "In other words: It is preferable having fewer users, who all know they should cite, over having many users, who do not know they should cite. If the goal had been to get more users, then the license would have been public domain. ... The citation notice is about (indirect) funding - nothing else." Does that fit with your assu…

What kind of “funding” is referenced in the FAQ? Is there some kind of organization that I am not aware of that pays the author(s) for citations in papers? How is the “long term survival” impacted by whether the author receives citations? I’m confused as to how “[not including citations] would not have been sustainable in the long term” unless either citations become money at some point or the author is motivated suf…

Isn't that explained in the link in the very first question?

https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...

Re: GNU Parallel Cheat Sheet [pdf]

#49

Ah, the rare case of nagware in GNU. From the man page: "--citation Print the BibTeX entry for GNU parallel and silence citation notice. If it is impossible for you to run --bibtex you can use --will-cite. If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do no…

Beyond being supremely irritating, nagware is simply not scalable. Imagine if every utility, library, or driver in a typical Linux distribution took this approach. :( I encourage Debian et al. to adopt a "no nagware" policy.

I am trying to understand your take. Do you also call Firefox nagware, when it pops up with a dialog box where you can click "Don't show this again"?

To me the dialog box is actually worse, because the program often blocks until you close the dialog box (not 100% sure if that is the case with Firefox).

With GNU Parallel you run 'parallel --citation' once, and you are done. We are talking an effort of 15 seconds or less.

When I install a library I often have to run the install command and it often takes longer than 15 seconds.

Finally, I would like to understand why you do not just use another utility? Would that not solve your issue?

Re: GNU Parallel Cheat Sheet [pdf]

#50

Earlier quoted context omitted.

Do you feel this is covered in the FAQ? http://git.savannah.gnu.org/cgit/parallel.git/plain/doc/cita...

The FAQ makes it worse: "GNU Parallel is indirectly funded through citations. It is therefore important for the long term survival of GNU Parallel that it is cited. The citation notice makes users aware of this." It's a bit like saying: "Webkit is indirectly funded by iPhones. It is therefore important for the long term survival of Webkit that people purchase iPhones. The iPhone notice make users aware of this."

We probably have to agree to disagree on that one:

I see loads of commercials for buying iPhones. I do not see a lot of commercials for citing GNU Parallel.

If I have to make a citation, it will cost me no money, but one line of text if I write an article. If I have to buy an iPhone, it will cost me many hours of work.

To me the two things are not even close to be similar.

But I can find one similar aspect: No one forces you to use an iPhone.

Post reply on HN