Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

71–80 of 277 posts

Re: GNU Parallel, where have you been all my life?

#71

Post by cnr »

dsissitka wrote: post #58
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

It's in current Fedora's: [david@pc ~]$ echo foo | parallel echo Academic tradition requires you to cite works you base your article on. If you use programs that use GNU Parallel to process data for an article in a scientific publication, please cite: Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин'). Zenodo. https://doi.org/10.5281/zenodo.8175685 This helps funding further development; AND IT WON'T COST…

Does 'Приго́жин' relate somehow to infamous 'Евгений Пригожин'. Or am I just biased?

Re: GNU Parallel, where have you been all my life?

#72

Post by 542458 »

Matl wrote: post #68
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> And calling that GPL? You can add any message you want into your GPL program. Also, a GPL program does not have to be free. This has nothing to do with the GPL. You can say in your program that 'by using this software you agree that you're a a cat' and license it under the GPL. That does not mean the GPL relates to cats in any way.

You can add all the extra restrictions you want, but they effectively won’t do anything. Expecting both the GPL and the additional restrictions to apply is a violation of section 7 of the GPL.

> All other non-permissive additional terms are considered “further restrictions” within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term.

Debian explains this further in their patch file.

Re: GNU Parallel, where have you been all my life?

#74

Post by imajoredinecon »

bloopernova wrote: post #43
amelius wrote: post #35

Another reminder that you shouldn't use Bash to write scripts. E.g. in Python this would all be very easy to do. Just start a bunch of threads and e.g. invoke subprocess.run() from them.

You don't have to reinvent the wheel for your script, all the parallel options are ready for you to use and are well documented. It's also packed with features that might take a long time to write into your Python script. I am trying to use Python by default when writing scripts nowadays, but sometimes the best tool for the job isn't Python or writing your own Python.

IMO, effective "scripting" just means the ability to solve ad hoc problems easily by writing task-specific glue that delegates the hard parts of the program to (1) an effective set of libraries you've written yourself and (2) external code or tools when it makes sense.

From this perspective, the languages of the glue, the libraries, and the external code all matter less than the ease of writing the glue; interfacing with the external code; and maintaining the libraries. The best language for this probably comes down to a combination of what you're comfortable writing (and reading, and maintaining) and what kinds of tasks you're trying to solve.

For me personally, using Python glue and libraries strikes a pretty good balance here. Writing a script "in Python" doesn't mean you need to reinvent the wheel. If you think `parallel` provides a better interface for map-reduce parallelism than `subprocess` (or than a library function you've written on top of `subprocess`), no problem: you can just call `parallel` from Python (and you'll probably find yourself writing a library function on top of it to abstract away the fact that it's a shell script).

But if you're much more effective working in Bash than Python, then writing your glue and developing your libraries in Bash could be the way to go.

Re: GNU Parallel, where have you been all my life?

#75

Post by bduffany »

cusspvz wrote: post #53

You guys know that in bash you can use `&` to pass a foreground terminal process to the background and then use `wait` to wait for all the session's background process to end, right?

Yes, and those work well for smaller workloads, but if you just run 1,000,000 commands with `&` in a `for` loop, it will grind your computer to a halt (if the tasks are modestly resource intensive). GNU parallel will let you run those same 1,000,000 tasks but make sure that only (e.g.) 16 of them are running at once. It's not easy to do that in bash.

Re: GNU Parallel, where have you been all my life?

#76

Post by em500 »

bloopernova wrote: post #44
em500 wrote: post #32

Earlier quoted context omitted.

Except Perl isn't always present by default either (e.g. in Arch Linux or FreeBSD).

parallel --embed > parallel.sh Then store that in your source repo and use it wherever shells are used!

On Debian 11.7:

   $ parallel --embed > parallel.sh
   Unknown option: embed
[edit] Ran it in Ubuntu 22.04, it does output a bash script ... which still depends on Perl.

Re: GNU Parallel, where have you been all my life?

#78

Post by akerl_ »

Matl wrote: post #68
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> And calling that GPL? You can add any message you want into your GPL program. Also, a GPL program does not have to be free. This has nothing to do with the GPL. You can say in your program that 'by using this software you agree that you're a a cat' and license it under the GPL. That does not mean the GPL relates to cats in any way.

Can you back that up?

If I put the GPL in my software and add a file next to it that says "Also you can't use this software if you make more than $100k/year", I've pretty clearly added an additional clause that's incompatible with the GPL.

Re: GNU Parallel, where have you been all my life?

#79

Post by ilyt »

5e92cb50239222b wrote: post #41
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

I wanted to say 'not anymore', but it turns out that some distributions remove that message. https://gitlab.archlinux.org/archlinux/packaging/packages/pa... Debian too (thanks to iib for pointing this out) https://salsa.debian.org/med-team/parallel/-/tree/master/deb... And looks like the author is aware of both: https://gitlab.archlinux.org/archlinux/packaging/packages/pa...

I kinda want to submit patch to GNU Parallel that changes "Hall of Shame" to "Hall of Heroes".

But yeah, if guy wants to have the name of the app mentioned there is a BSD license for that I thin...

Post reply on HN