GNU Parallel Cheat Sheet [pdf]

Post by arbie » Fri, Mar 08, 2019, 4:57 AM UTC

akramer wrote: post #41

Each time I've seen something about GNU parallel pop up I've been tempted to post, but I've never made an account until now. I wrote a very different style of command parallelizer that I named lateral. It doesn't require constructing elaborate commandlines that define all of your work at once. You start a server, and separate invocations of 'lateral run' add your commands to a queue to run on the server, including th…

Post by the_it_girl » Fri, Mar 08, 2019, 12:20 PM UTC

akramer wrote: post #41

Each time I've seen something about GNU parallel pop up I've been tempted to post, but I've never made an account until now. I wrote a very different style of command parallelizer that I named lateral. It doesn't require constructing elaborate commandlines that define all of your work at once. You start a server, and separate invocations of 'lateral run' add your commands to a queue to run on the server, including th…

Post by jwilk » Fri, Mar 08, 2019, 4:09 PM UTC

rurban wrote: post #32

LeoPanthera wrote: post #28

Earlier quoted context omitted.

It would be trivial to fork parallel. If people cared enough, a fork would appear and be adopted. That's the beauty of free software. If you don't like it you can change it.

It was already forked in its early perl days, and thus it's very hard to use it properly as build tool, as the non-GNU version has an entirely different argument syntax. eg on macOS or BSD. You really have to probe for the GNU version (the more popular and newer one, with this awkward citation and begging), but for longer tasks it speeds up processing immensely. There's no need for Hadoop when you can use parallel. I…

Post by rurban » Fri, Mar 08, 2019, 5:17 PM UTC

jwilk wrote: post #55

rurban wrote: post #32

Earlier quoted context omitted.

It was already forked in its early perl days, and thus it's very hard to use it properly as build tool, as the non-GNU version has an entirely different argument syntax. eg on macOS or BSD. You really have to probe for the GNU version (the more popular and newer one, with this awkward citation and begging), but for longer tasks it speeds up processing immensely. There's no need for Hadoop when you can use parallel. I…

Where can I read about this fork? Googling for "bsd parallel command" doesn't seem to show anything relevant.

    dnl GNU parallel, skip the old non-perl version from moreutils so far
    AC_CHECK_PROGS([PARALLEL], [parallel])
    logs_all=logs-all-serial.sh.in
    if test -n "$PARALLEL"; then
      AC_MSG_CHECKING([PARALLEL version])
      parallel_version=`$PARALLEL --version 2>&1 | head -n1 | cut -c14-`
      case "$parallel_version" in
        [0-9]*) AC_MSG_RESULT([$parallel_version])
                logs_all=logs-all-parallel.sh.in
                ;;
        *invalid*)
                PARALLEL=
                parallel_version="skip old moreutils version, need GNU parallel"
      esac
      AC_MSG_RESULT([$parallel_version])
    fi
    AM_CONDITIONAL([HAVE_PARALLEL], [test -n "$PARALLEL"])

Post by oids98d98s » Fri, Mar 08, 2019, 6:08 PM UTC

oids98d98s wrote: post #12

bonoboTP wrote: post #11

Ah, the rare case of nagware in GNU. From the man page: "--citation Print the BibTeX entry for GNU parallel and silence citation notice. If it is impossible for you to run --bibtex you can use --will-cite. If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU parallel is indirectly financed through citations, so if your users do no…

Gooood who caaaaares. They spent a lot of time and effort, and made a cool thing and gave it away for free. If it bothers you so much, just add the flag. Or patch it out.