How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#81

Post by sirn » Sun, Jul 07, 2019, 5:40 PM UTC

dymk wrote: post #43

mooneater wrote: post #41

Great article, but a common scenario is missing: I want to run a long running script, but not if another copy of it is already running.

https://linux.die.net/man/1/flock would probably help

flock(1) is not POSIX, though. mkdir(1) can be used if you absolutely want a POSIX way to manage locks. For example:

    if ! mkdir .lock; then
        printf >&2 "Already running?\\n"
        exit 1
    fi

Some network file system implementation do not guarantee atomic mkdir, so you still need an extra caution with this method.

Re: How to write idempotent Bash scripts

#82

Post by aey » Sun, Jul 07, 2019, 5:47 PM UTC

andreareina wrote: post #17

aey wrote: post #3

Why wouldn’t you use make? Side-effect tracking and cleanup is the job of a build tool.

How do you get make to track something other than the existence and age of a file?

If it’s IO, then the inputs and outputs can be tracked as files. If it’s not IO, and just a pure computation it can be reproduced on demand.

Re: How to write idempotent Bash scripts

#83

Post by pletnes » Sun, Jul 07, 2019, 5:59 PM UTC

rb808 wrote: post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

If you’re into minimal docker images, bash or similar shell might be the only script interpreter available.

Python might not exist on your windows machine.

Etc.

Re: How to write idempotent Bash scripts

#84

Post by snissn » Sun, Jul 07, 2019, 6:03 PM UTC

farslan wrote: post #76

dymk wrote: post #38

Earlier quoted context omitted.

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Op here. Exactly this. I’ve seen many times where people had scripts and had to modify it (such as commenting code) when something has failed in the middle of the script.

Don't forget about pipefail!!

set -eu set -o pipefail

Anything else I'm missing?

Re: How to write idempotent Bash scripts

#85

Post by rectangletangle » Sun, Jul 07, 2019, 6:06 PM UTC

pmarreck wrote: post #58

rgoulter wrote: post #56

Earlier quoted context omitted.

FWIW, I don't think "hermetic" is in widespread usage. My understanding is that "hermetic builds" refer to the kind of thing that tools like Guix, Nix, etc. go for. Something like "Declare all the inputs and dependencies to get a reproducible output".

I prefer "deterministic" in this case (I think?)

I've always heard "deterministic" or "reproducible," curious if the term "hermetic" is somehow different in this context.

Re: How to write idempotent Bash scripts

#86

Post by heavenlyblue » Sun, Jul 07, 2019, 6:08 PM UTC

dymk wrote: post #53

heavenlyblue wrote: post #48

Earlier quoted context omitted.

Rust would help due to the type system and lack of library dependencies (it does static linking AFAIK), I suppose.

I feel like my comment went in one eyeball and out the other. Chef and friends are blocked on disk I/O. How does a typesystem and/or thinner abstraction layer to disk I/O speed up the underlying expensive operation: blocking disk I/O?

I - on the other hand, feel like your comment somehow forgot that the op said “safety, speed and convenience”, so you attacked the least important point made in the first place.

It’s much easier to see the failure conditions in a Rust program rather than in bash. Also rust seems like easier to maintain, too.

Re: How to write idempotent Bash scripts

#87

Post by RijilV » Sun, Jul 07, 2019, 6:10 PM UTC

kousikk wrote: post #35

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Last modified timestamps in source code are a bad idea and they make builds non-hermetic. Better to fix up those scenarios in code than work around them. Irrespective you have a good point that touch does have side effects.

I hate to nitpick, but from the man page of touch:

> touch -- change file access and modification times

If one were to argue side effects of touch, it would be that non-existent files are created. The purpose of touch is to update access and modification times.

Re: How to write idempotent Bash scripts

#88

Post by eridius » Sun, Jul 07, 2019, 6:14 PM UTC

pmarreck wrote: post #58

rgoulter wrote: post #56

Earlier quoted context omitted.

FWIW, I don't think "hermetic" is in widespread usage. My understanding is that "hermetic builds" refer to the kind of thing that tools like Guix, Nix, etc. go for. Something like "Declare all the inputs and dependencies to get a reproducible output".

I prefer "deterministic" in this case (I think?)

"deterministic" just means with a given set of inputs you get the same bitwise-identical output each time.

Re: How to write idempotent Bash scripts

#89

Post by khc » Sun, Jul 07, 2019, 6:15 PM UTC

snissn wrote: post #84

farslan wrote: post #76

Earlier quoted context omitted.

Op here. Exactly this. I’ve seen many times where people had scripts and had to modify it (such as commenting code) when something has failed in the middle of the script.

Don't forget about pipefail!! set -eu set -o pipefail Anything else I'm missing?

nounset

Re: How to write idempotent Bash scripts

#90

Post by pdkl95 » Sun, Jul 07, 2019, 6:20 PM UTC

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

> traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state

Traps are underused. They solve a lot of error-handling and cleanup problems really easily!

Here[1] is a simple example of using a trap to cleanup a tempfile, that safely handles pipeline errors and unexpected exit/return. The same basic idea works for many kinds of error handling, cleanup, or similar work.

If you ever need to cleanup something at the end of a function

    foo() {
        setup_command
        # ...
        cleanup_command
    }

it using a trap might be as simple as moving the cleanup command to trap at the earliest possible time in the function the command can run:

    foo() {
        setup_command
        trap "cleanup_command" RETURN
        # ...
    }

[1] https://gist.github.com/pdkl95/61fc242e7961cc2584a787ed1760c...