GNU Parallel, where have you been all my life? • Hacker News Board

Re: GNU Parallel, where have you been all my life?

#101

Post by foldr » Mon, Aug 21, 2023, 1:53 PM UTC

dahart wrote: post #81

throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> it’s not tradition to write citations for tools used to conduct research

Academics seem to have a very blinkered attitude to this. I wrote some software that was popular for a while in a niche field, and people were forever asking me to waste my time by 'publishing' the manual in some pointless journal so that they could cite something and give me credit. Writing useful software counts for less in that world than publishing another pointless paper that no-one will read.

Re: GNU Parallel, where have you been all my life?

#102

Post by toastal » Mon, Aug 21, 2023, 1:54 PM UTC

ssddanbrown wrote: post #14

Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick: find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}

You should batch compress to JPEG XL too with cjxl --lossless_jpeg=1 --quality=80 --effort=9 {} {/.}.jxl (or magick)

Re: GNU Parallel, where have you been all my life?

#103

Post by dpkirchner » Mon, Aug 21, 2023, 1:55 PM UTC

bloopernova wrote: post #44

em500 wrote: post #32

Earlier quoted context omitted.

Except Perl isn't always present by default either (e.g. in Arch Linux or FreeBSD).

parallel --embed > parallel.sh Then store that in your source repo and use it wherever shells are used!

Would this taint the other code in your repo with the GPL? I'd guess it would depend on how it is distributed.

Re: GNU Parallel, where have you been all my life?

#104

Post by tempaccount420 » Mon, Aug 21, 2023, 1:55 PM UTC

throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

Why not just cite it?

Re: GNU Parallel, where have you been all my life?

#105

Post by flanked-evergl » Mon, Aug 21, 2023, 1:56 PM UTC

dahart wrote: post #81

throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> IIRC the citation notice was cleared by Stallman as GPL.

I really hope that whomever adjudicate these disputes regarding licence agreements doesn't care what a random person says about it.

Re: GNU Parallel, where have you been all my life?

#106

Post by DiggyJohnson » Mon, Aug 21, 2023, 2:02 PM UTC

flanked-evergl wrote: post #105

dahart wrote: post #81

Earlier quoted context omitted.

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> IIRC the citation notice was cleared by Stallman as GPL. I really hope that whomever adjudicate these disputes regarding licence agreements doesn't care what a random person says about it.

It's disingenuous to say that Richard Stallman is a "random person" regarding the GPL.

Re: GNU Parallel, where have you been all my life?

#107

Post by BoppreH » Mon, Aug 21, 2023, 2:04 PM UTC

It's a nice tool, but it also shows the shortcomings of shell commands.

In a proper programming language, we'd have something like

    parallel [1..5], i => { sleep random()*10+5; possibly_flaky i }
    // [{"Seq": 4, "Host": ":", "Starttime": 1692491267...

And `parallel` would only have to worry about parallelization.

Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating format ({1}), and a way to output a list of structures (CSV-like). You can see the same issues in `find`, where the exec separator is `\;`, the template is `{}`, and the output is delimited by \n or \0. And `xargs` does it in yet another different way.

It's very hard to acquire and retain mastery over a toolbox where every tool reinvents the basics. If you ever found yourself searching "find exec syntax" multiple times in a week, it's not your fault.

As for alternatives, I'm a fan of YSH[1] (Javascript-like), Nushell[2] (reinvented from first-principles for simplicity and safety) and Fish[3] (bash-like but without the footguns). Nushell is probably my favorite from the bunch, here's a parallel example:

    ls | where type == dir | par-each { |it|
        { name: $it.name, len: (ls $it.name | length) }
    }

[1] https://www.oilshell.org/release/latest/doc/ysh-tour.html

[2] https://github.com/nushell/nushell

[3] https://fishshell.com/

Re: GNU Parallel, where have you been all my life?

#108

Post by OJFord » Mon, Aug 21, 2023, 2:08 PM UTC

bloopernova wrote: post #44

em500 wrote: post #32

Earlier quoted context omitted.

Except Perl isn't always present by default either (e.g. in Arch Linux or FreeBSD).

parallel --embed > parallel.sh Then store that in your source repo and use it wherever shells are used!

To install parallel, first run parallel --embed?

Re: GNU Parallel, where have you been all my life?

#109

Post by computerfriend » Mon, Aug 21, 2023, 2:08 PM UTC

cnr wrote: post #71

dsissitka wrote: post #58

Earlier quoted context omitted.

It's in current Fedora's: [david@pc ~]$ echo foo | parallel echo Academic tradition requires you to cite works you base your article on. If you use programs that use GNU Parallel to process data for an article in a scientific publication, please cite: Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин'). Zenodo. https://doi.org/10.5281/zenodo.8175685 This helps funding further development; AND IT WON'T COST…

Does 'Приго́жин' relate somehow to infamous 'Евгений Пригожин'. Or am I just biased?

Yeah, I think it does.

Re: GNU Parallel, where have you been all my life?

#110

Post by pneumic » Mon, Aug 21, 2023, 2:08 PM UTC

cnr wrote: post #71

dsissitka wrote: post #58

Earlier quoted context omitted.

It's in current Fedora's: [david@pc ~]$ echo foo | parallel echo Academic tradition requires you to cite works you base your article on. If you use programs that use GNU Parallel to process data for an article in a scientific publication, please cite: Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин'). Zenodo. https://doi.org/10.5281/zenodo.8175685 This helps funding further development; AND IT WON'T COST…

Does 'Приго́жин' relate somehow to infamous 'Евгений Пригожин'. Or am I just biased?

Prigogine/Prigozhin is a not-uncommon Russian-Jewish name: https://en.wikipedia.org/wiki/Prigozhin