Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

231–240 of 277 posts

Re: GNU Parallel, where have you been all my life?

#231

Post by dahart »

paulmd wrote: post #221
dahart wrote: post #181

Earlier quoted context omitted.

That’s not true. The language of Parallel’s citation notice, while confusing to some users, does not impose any legal requirements and is not part of the license. Neither the notice nor the license claim otherwise. RMS, and more importantly, Ole Tange, agree that Parallel’s notice is not legally binding, and intended to write it that way, and there is a publicly visible history of this intention and agreement.

I don't think anyone would make a deal with oracle on a "don't worry this isn't legally binding, you're just stating your intent to comply" basis.

Indeed, and people are choosing not to use Parallel for the same reason. The notice would be much better IMO from the user perspective if it was more clear. I guess that’s maybe the point, to leave people with the mistaken impression that this is a binding agreement.

Re: GNU Parallel, where have you been all my life?

#232

Post by cb321 »

lysium wrote: post #147
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

On Unix, you might try Nim (https://nim-lang.org) with https://github.com/Vindaar/shell and there are a slew of pipeline-y/Unix-y utilities over at https://github.com/c-blake/bu

Nim is statically typed and (generally) native-compiled, but it has very low ceremony ergonomics and a powerful compile-time macro/template system as well as user-defined operators (e.g., you can use `+-` to make a constructor for uncertain values so that `9 +- 2` builds a typed object as in https://github.com/SciNim/Measuremancer .

Re: GNU Parallel, where have you been all my life?

#233

Post by steveklabnik »

reddit_clone wrote: post #218
chasil wrote: post #171

Earlier quoted context omitted.

GNU Parallel is also based on perl, so the footprint is quite large. GNU xargs implements limited parallelization, and is compiled C. This functionality is present within busybox, including the Windows version. https://www.linuxjournal.com/content/parallel-shells-xargs-u... GNU Parallel will have much greater functionality, but it will not reach as far as xargs.

> GNU Parallel is also based on perl Time to rewrite it in Rust /s :p

There have been multiple ports already, I believe.

Re: GNU Parallel, where have you been all my life?

#234

Post by memefrog »

dahart wrote: post #187
memefrog wrote: post #173

Earlier quoted context omitted.

>His opinion wasn’t necessary, but since he wrote the license, it is authoritative. No it isn't. Licences, like most legal documents, are construed objectively. The subjective intention of the author is totally irrelevant to the meaning.

You might have misunderstood what I said. It’s not up for debate whether RMS’s opinions or intent on the GPL have affected industry practice; that’s a fact of history. His statements on the GPL are authoritative in the sense that they may have prevented the courts from examining this question.

"Authoritative" has a particular meaning. You might have intended to say "influential". You didn't. I can only reply based on what you said.

Re: GNU Parallel, where have you been all my life?

#235

Post by paulddraper »

lysium wrote: post #147
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

Python.

Just the inclusion of argparse alone is worth it IMO.

> Python’s support for pipes was difficult

Well, the idea would be to replace a lot of your pipe usage.

Off the wall, but Scala has a concise syntax for process operations, but startup time is likely prohibitive.

Re: GNU Parallel, where have you been all my life?

#236

Post by memefrog »

dahart wrote: post #181
memefrog wrote: post #174

Earlier quoted context omitted.

>in this case Parallel’s notice is not a modification of the license at all This is a question of law that only a court can answer.

That’s not true. The language of Parallel’s citation notice, while confusing to some users, does not impose any legal requirements and is not part of the license. Neither the notice nor the license claim otherwise. RMS, and more importantly, Ole Tange, agree that Parallel’s notice is not legally binding, and intended to write it that way, and there is a publicly visible history of this intention and agreement.

RMS, not being a judge, is incapable of "authoritatively" or otherwise determining whether this notice is legally binding.

If it is something that needs to be "confirmed" by someone "authoritatively" then you should ask a lawyer for advice. You should not ask a programmer for a "ruling".

What RMS might be saying is "we won't seek to enforce it". That is completely different.

Re: GNU Parallel, where have you been all my life?

#237

Post by c-hendricks »

wiredfool wrote: post #40
ssddanbrown wrote: post #14

Love finding a good use-case of parallel as an easy way to gain massive time savings, especially on the modern high-threaded CPUs of today. Most recently found it useful when batch-compressing large jpeg images to smaller webp files, via use with find and ImageMagick: find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {}

Xargs is a nearly drop in replacement and probably already installed by default in most distros. You may need the -n 1 (one file per) and -P to parallelize. xargs -n 1 -P 8

find + xargs has become my go-to "process files in parallel". Tho now I'm wondering if I should be using `-n` instead of `-L`

    #!/usr/bin/env bash
    set -e

    main() {
      if [ "$1" = "handle-file" ]; then
        shift
        handle-file "$@"
      else
        find . \
          -type f \
          -not -path '*/optimized/*' \
          -print0 \
          | xargs \
            -0 \
            -L 1 \
            -P 8 \
            -I {} \
            bash -c "cd \"$PWD\" && \"$0\" handle-file \"{}\""
      fi
    }

    handle-file() {
      echo "handle-file $1 ..."
    }

    main "$@"

Re: GNU Parallel, where have you been all my life?

#238

Post by b5n »

jooz wrote: post #179

I try to use it last week to run 10 instances of curl against a webserver. I was expecting something simple as 'parallel -j10 curl https://whatever ' but couldnt find the right syntax in less time that took me to prepare a dirty shell script that did the same.

Quick solution:

    parallel -j 10 curl 2> /dev/null \
        ::: $(for i in {1..10};do echo 'https://whatever.com';done)

Re: GNU Parallel, where have you been all my life?

#239

Post by Symbiote »

foldr wrote: post #122
Symbiote wrote: post #115

Earlier quoted context omitted.

A more useful option is to use Zenodo to provide a DOI for a GitHub repository. https://docs.github.com/en/repositories/archiving-a-github-r...

That doesn't really help. People already know how to paste the URL for a piece of software into a paper. It's more that it doesn't count for anything (because it's a piece of software and not a paper).

It does help, as the DOI system provides the tracking needed to count citations and measure the "effectiveness" of the researcher.

Re: GNU Parallel, where have you been all my life?

#240

Post by gorjusborg »

thomasahle wrote: post #223
gorjusborg wrote: post #132

Earlier quoted context omitted.

> They get no citations and eventually have to leave academia. You're welcome? Seriously though, adding the citation nag to software is two wrongs not making a right. As a software user, it isn't my fault academia hasn't figured out how to reward software contribution. If they can't figure it out, finding a greener pasture makes a lot of sense.

> As a software user, it isn't my fault academia hasn't figured out how to reward software contribution. If you're not writing papers, the citation notification isn't for you. Can't you just mute it and continue using the software without worries?

> Can't you just mute it and continue using the software without worries?

It _seems_ like a reasonable thing to ask, it's a minor inconvenience, really, so what's the big deal?

The big deal is that the behavior doesn't fit the unix philosophy. Tools are meant to do one thing, and do it well. They get composed in pipelines to get jobs done. In these pipelines, the communication medium is text, via stdin/stdout/stderr. If a tool is unpredictable in what it puts out via text, it can make the whole pipeline unpredictable, or at least more complicated.

If it _was_ okay, we should welcome everyone putting nag features in these simple cli tools, right? Well, I'd be on board with that as long as I can blanket disable all of them. If not, let's just leave our political/professional/begging messaging outside our computing tools. Okay?

Post reply on HN