Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

Post reply on HN
Print view View original

11–20 of 195 posts

Re: How to write idempotent Bash scripts

#11

Post by mnutt »

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef will determine if the directory already exists.

There are of some downsides to that approach which stem from the fact that you can’t possibly declare every possible thing. For instance, if you were to run the above declaration and then later remove the declaration, chef wouldn’t know to delete the directory; at that point chef wouldn’t know anything about the directory at all. So ideally you can build infrastructure from scratch each time (Docker, etc) rather than having to converge an existing machine.

Re: How to write idempotent Bash scripts

#12

Post by itchyouch »

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Chef does have an action on their primitives, so it is possible in the directory example to set the action to :delete instead of only implying :create

Re: How to write idempotent Bash scripts

#13

Post by NelsonMinar »

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Re: How to write idempotent Bash scripts

#14

Post by intc »

> This is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects

Every call to touch will alter the modification time of the file (example.txt in this case).

Re: How to write idempotent Bash scripts

#15

Post by mongol »

This is useful and I think it would be good if more tools had idempotency options available. Just today I needed to run a script that added a zypper repo in OpenSuse. It fails the second time because the repo is already added and there is no option to avoid it. Also the error code is not distinct so it is not easy to script around. Ideally every tool should have a mode "accomplish this result" in addition to "do this".

Re: How to write idempotent Bash scripts

#18

Post by jrumbut »

intc wrote: post #14

> This is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects Every call to touch will alter the modification time of the file (example.txt in this case).

It will change the access time too. Almost every suggestion in the post has this problem which is unfortunate since access/modify/create time are used in a lot of scripts to see which files were or were not processed by prior runs of a script (whether this is a good idea or not).

It is very difficult to write truly idempotent bash scripts (consider log files). Creating sort of idempotent or at least rerunnable scripts would be nice but I think even that would take more care than this.

Re: How to write idempotent Bash scripts

#19

Post by cjvirtucio »

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.
Post reply on HN