Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

121–130 of 277 posts

Re: GNU Parallel, where have you been all my life?

#121

Post by Eiim »

dahart wrote: post #81
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

Citing tools is maybe not tradition, but it's gaining popularity. R, for example, has 320,000 cites. (https://scholar.google.com/scholar?cites=1605575084878280829...)

Re: GNU Parallel, where have you been all my life?

#122

Post by foldr »

Symbiote wrote: post #115
foldr wrote: post #101

Earlier quoted context omitted.

> it’s not tradition to write citations for tools used to conduct research Academics seem to have a very blinkered attitude to this. I wrote some software that was popular for a while in a niche field, and people were forever asking me to waste my time by 'publishing' the manual in some pointless journal so that they could cite something and give me credit. Writing useful software counts for less in that world than p…

A more useful option is to use Zenodo to provide a DOI for a GitHub repository. https://docs.github.com/en/repositories/archiving-a-github-r...

That doesn't really help. People already know how to paste the URL for a piece of software into a paper. It's more that it doesn't count for anything (because it's a piece of software and not a paper).

Re: GNU Parallel, where have you been all my life?

#123

Post by folmar »

jnsaff2 wrote: post #89
5e92cb50239222b wrote: post #66

Earlier quoted context omitted.

> Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин'). Looks like the latest release is named after Prigozhin. Yeah, probably that one, although I couldn't find anything in the mailing list to confirm it. https://en.wikipedia.org/wiki/Yevgeny_Prigozhin edit: all releases are named after current political events: https://git.savannah.gnu.org/cgit/parallel.git/refs/

Looking at the release history it may be that the releases are named after some front page daily news headline or something. Even so, knowingly naming a release after a war criminal is very off-putting.

It shall not be. It is a notability indication, not endorsement, just take a look at other release names from the time of the war. Or compare to Time's "Person of the year".

Re: GNU Parallel, where have you been all my life?

#124

Post by seized »

cusspvz wrote: post #53

You guys know that in bash you can use `&` to pass a foreground terminal process to the background and then use `wait` to wait for all the session's background process to end, right?

And that's not really at all comparable to what Parallel can do.... Bash can't do that across thousands of cores on separate machines for example.

Re: GNU Parallel, where have you been all my life?

#125

Post by commonlisp94 »

em500 wrote: post #76
bloopernova wrote: post #44

Earlier quoted context omitted.

parallel --embed > parallel.sh Then store that in your source repo and use it wherever shells are used!

On Debian 11.7: $ parallel --embed > parallel.sh Unknown option: embed [edit] Ran it in Ubuntu 22.04, it does output a bash script ... which still depends on Perl.

isn't perl always installed?

Re: GNU Parallel, where have you been all my life?

#126

Post by aftbit »

gjvc wrote: post #55
green-orca wrote: post #34

I'm using task spooler a lot for parallel background processing. What I like the most it the ability to add further tasks to the queue after processing has already started. https://manpages.ubuntu.com/manpages/xenial/man1/tsp.1.html

from that man page, there is a name clash with "ts" from moreutils

moreutils also clashes with parallel, does it not? i remember installing some package for chronic and thus breaking GNU parallel, at least back in the late 2010s.

Re: GNU Parallel, where have you been all my life?

#127

Post by coliveira »

BoppreH wrote: post #107

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Re: GNU Parallel, where have you been all my life?

#128

Post by rlpb »

dahart wrote: post #81
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> and the citation notice is a one-time thing you can silence permanently

This doesn't scale. Imagine if all the software you used nagged you and had their own individual methods to silence them. I don't think this would be reasonable.

What makes this particular software so special?

Re: GNU Parallel, where have you been all my life?

#129

Post by adrian_b »

em500 wrote: post #32
ketanmaheshwari wrote: post #10

Earlier quoted context omitted.

The entirety of GNU Parallel is just one Perl program. It could be copied over and used in a pinch. The installation itself is very simple and no special dependencies or privileges are needed.

Except Perl isn't always present by default either (e.g. in Arch Linux or FreeBSD).

There are also many Linux distributions that do not install by default all the POSIX utilities, but only the minimal set that is needed to bootstrap the system.

On all such systems, it is very easy for the user to install any missing POSIX utility, but it is also easy to install any non-POSIX GNU utility.

So not even xargs is certain to exist by default on all systems.

Moreover, POSIX xargs is restricted to execute sequentially all processes.

Any use of xargs for parallel execution is non-POSIX, so in that case there is no reason to not use "parallel" instead.

Re: GNU Parallel, where have you been all my life?

#130

Post by dahart »

herrkanin wrote: post #98

I have wanted to parallelize my .zshrc file for a while – all those environment setup scripts for nvm, pyenv, starship, etc really makes the startup time noticably slow. Does anyone know how to do this?

Ooh nice thought. I’m not certain, but I kinda doubt it’s possible, because those startup scripts need to modify the current shell environment. I believe GNU parallel runs in a subshell and launching new tasks in separate processes, so fundamentally doesn’t operate the same way that e.g. sourcing the nvm script does, unfortunately. Even if there was some way to hack it, I’d be nervous about changing environment variables in parallel, to me that sounds like asking for really nasty race condition bugs.
Post reply on HN