Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design of GNU Parallel (2015)

gnu.org

Post reply on HN
Print view View original

11–20 of 75 posts

Re: Design of GNU Parallel (2015)

#11

Post by KronisLV »

michalc wrote: post #9

I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications. And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start anothe…

I've used Supervisor pretty successfully for this as well: http://supervisord.org/

Example of installing it in a Debian/Ubuntu container during container build, here's an example Dockerfile:

  RUN apt-get update \
      && apt-get -yq --no-upgrade install \
          supervisor \
      && apt-get clean \
      && rm -rf /var/lib/apt/lists /var/cache/apt/*
Then it's possible to create a configuration file, for example /etc/supervisord.conf, to specify what should run and how:

  [supervisord]
  nodaemon=true
  
  [program:php-fpm]
  command=/usr/sbin/php-fpm8.0 -c /etc/php/8.0/fpm/php-fpm.conf --nodaemonize
  stdout_logfile=/dev/stdout
  stdout_logfile_maxbytes=0
  stderr_logfile=/dev/stderr
  stderr_logfile_maxbytes=0
  
  [program:nginx]
  command=/usr/sbin/nginx
  stdout_logfile=/dev/stdout
  stdout_logfile_maxbytes=0
  stderr_logfile=/dev/stderr
  stderr_logfile_maxbytes=0
And finally it can be run inside of the container entrypoint, along the lines of this in docker-entrypoint.sh:

  #!/bin/bash
  echo "Software versions..."
  nginx -V && supervisord --version
  
  echo "Running Supervisor..."
  supervisord --configuration=/etc/supervisord.conf
Here's more information about the configuration file format, in case anyone is curious: http://supervisord.org/configuration.html

It should be noted that this package will bring in some dependencies, though, which may or may not be okay, depending on how stringent you are about space usage and what's in your containers, example for a Ubuntu container:

  The following NEW packages will be installed:
    libexpat1 libmpdec3 libpython3-stdlib libpython3.10-minimal libpython3.10-stdlib libreadline8 libsqlite3-0 media-types
    python3 python3-minimal python3-pkg-resources python3.10 python3.10-minimal readline-common supervisor
  0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
  Need to get 6905 kB of archives.
  After this operation, 25.7 MB of additional disk space will be used.
(just found the piece of software itself useful for this use case, figured I'd share my experiences)

My problem is that it's not always immediately clear how software that would normally run as a systemd service could be launched in the foreground instead. It usually takes a bit of digging around.

Re: Design of GNU Parallel (2015)

#12

Post by RadiozRadioz »

psychphysic wrote: post #8
docandrew wrote: post #2

I couldn’t make heads or tails of what this would be useful for from the OP (maybe it’s something I should already have known), but this from the official site was pretty helpful: https://www.gnu.org/software/parallel/parallel_cheat.pdf

That cheat sheet is super enlightening! But quite useless as it'll print poorly and is overall a waste of resources to have that lovely beach scene in the background.

The beach will certainly make the cheat sheet stick in my memory, I can tell you that much.

Re: Design of GNU Parallel (2015)

#13

Post by noloblo »

hkt wrote: post #5
NortySpock wrote: post #4

I found GNU parallel useful when I wanted to queue up transcoding of flac files to mp3 on my Raspberry Pi. A few ffmpeg flags plus a list of files meant I could easily just saturate one job per core with a one-line bash command.

I've used it to parallelise updating hundreds of helm releases whose CI pipelines had ceased to exist. It is a neat tool.

Can you please share the example code in gnu parallel

Re: Design of GNU Parallel (2015)

#16

Post by ketzu »

This was quite interesting to look through!

Perl 5.8.0 is over 20 years old (https://dev.perl.org/perl5/news/2002/07/18/580ann/) while centOS 3.9 was released in 2007! At the same time it seems not-that-old and ancient.

My personal anecdote with gnu parallel was running into it while working in academia. It worked well and saved me some time, but I felt that it was unreasonable of a tool to ask for a citation to parallelise a script - it seemed that matplotlib, jupyter and co would need one as well. On the other hand, I decided to not use it, because I also feel that authors can ask for whatever they want.

Re: Design of GNU Parallel (2015)

#17

Post by michalc »

KronisLV wrote: post #11
michalc wrote: post #9

I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications. And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start anothe…

I've used Supervisor pretty successfully for this as well: http://supervisord.org/ Example of installing it in a Debian/Ubuntu container during container build, here's an example Dockerfile: RUN apt-get update \ && apt-get -yq --no-upgrade install \ supervisor \ && apt-get clean \ && rm -rf /var/lib/apt/lists /var/cache/apt/* Then it's possible to create a configuration file, for example /etc/supervisord.conf, to spe…

I have previously thought a bit about using something like Supervisor. And if I was running something a bit closer to the metal, with no other infrastructure to restart stuff, then I would be much more pro.

But if inside Docker when something else already has the job of restarting things if they fall over, then it feels a bit over complicated in that there are multiple ways of doing the restarting. Plus, I think there is a touch more visibility - it's all just command line arguments to parallel:

    parallel --will-cite --line-buffer --jobs 2 --halt now,done=1 ::: \
        "some_proc some args" \
        "another_proc some more args"

Re: Design of GNU Parallel (2015)

#19

Post by RhysU »

chungy wrote: post #6
cricalix wrote: post #3

parallel is a tool I've reached for many times; the citation bit it prints is odd - it seems to assume that the general use case is research/academic - but easily squelched. A sample use case would be having a file that has words in it, one per line, and you want to run a program that operates on each word (device name, dollar amount, whatever). Sure, you can use a loop, but if the words and actions are independent,…

Thankfully both Debian and Arch patch out the citation nonsense.

It is "nonsense" because...?

A) You don't understand. Please read the "Citation notice" section in the article.

B) You understand but don't use GNU Parallel.

C) You understand and use GNU Parallel in a non-academic setting and find the hassle of supplying --no-notice to be onerous vs the effort to write/maintain your own tool.

D) You understand and use GNU Parallel in an academic setting and have cited Ole or plan to cite Ole.

From the article, nearly 10 years ago Ole added the citation behavior after discussing it with his users: https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...

Ole's citations took off roughly coincident with this behavior being added: https://scholar.google.com/citations?hl=en&user=D7I0K34AAAAJ... (click "Cited By" and notice the bar chart).

Post reply on HN