How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#31

Post by vages » Sun, Jul 07, 2019, 3:58 PM UTC

joshspankit wrote: post #28

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

In other words: When promiscuous touching, always make sure you’re not making things unsanitary.

I'm sure you can find a way to avoid promiscuous touching if you read the (gentle)man page.

Re: How to write idempotent Bash scripts

#32

Post by bonyt » Sun, Jul 07, 2019, 3:59 PM UTC

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp.

https://linux.die.net/man/1/mktemp

Re: How to write idempotent Bash scripts

#33

Post by united893 » Sun, Jul 07, 2019, 4:01 PM UTC

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure.

Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe.

See:

> http://redsymbol.net/articles/unofficial-bash-strict-mode/

Re: How to write idempotent Bash scripts

#34

Post by stephenr » Sun, Jul 07, 2019, 4:02 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

I had the same thought about `ln -sf` - removing and recreating the symlink aren't no-effect.

Re: How to write idempotent Bash scripts

#35

Post by kousikk » Sun, Jul 07, 2019, 4:02 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Last modified timestamps in source code are a bad idea and they make builds non-hermetic. Better to fix up those scenarios in code than work around them.

Irrespective you have a good point that touch does have side effects.

Re: How to write idempotent Bash scripts

#36

Post by empath75 » Sun, Jul 07, 2019, 4:04 PM UTC

This is surely a good collection of flags to know, but I sort of object to the premise of the article.

There is a good reason that many of these flags are not default behavior, and it’s that they can be quite destructive.

If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state? Telling people to just always use these flags and not only when they’re quite sure what their script is doing, is probably playing with fire.

This may make the script idempotent in repeated runs, but the first run might not be what you expect at all.

Re: How to write idempotent Bash scripts

#37

Post by _0w8t » Sun, Jul 07, 2019, 4:09 PM UTC

stephenr wrote: post #34

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

I had the same thought about `ln -sf` - removing and recreating the symlink aren't no-effect.

To avoid that I use readlink in my scripts to read the previous value of the link and call ln -s only when the value is not what is expected.

Re: How to write idempotent Bash scripts

#38

Post by dymk » Sun, Jul 07, 2019, 4:13 PM UTC

united893 wrote: post #33

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`.

The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Re: How to write idempotent Bash scripts

#39

Post by dymk » Sun, Jul 07, 2019, 4:15 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Within the scope of the article, which is about modifying the structure and logical contents of the filesystem (a-la `chef converge` - the author never once mentions FS metadata), it's idempotent.

Re: How to write idempotent Bash scripts

#40

Post by eddieroger » Sun, Jul 07, 2019, 4:17 PM UTC

pas wrote: post #22

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

They didn't claim Chef was magic, and yeah it's just a bunch of cookbooks, but that doesn't change their point that it's declarative and useful for describing an end-state. Chef is hard and slow if you rely solely on others' work without understanding what it's doing, which is functionally equivalent to magic. But if you know how to tune things, keep your cookbooks down to only required ones, and a few other tricks you'll learn from usage, it can be fast as anything. I've seen one particular setup meant to keep a server up to date and configured to have HAProxy serve over 100+ endpoints run in under a minute. It's possible, just a different approach.