How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#21

Post by zokier » Sun, Jul 07, 2019, 3:32 PM UTC

The first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be

    echo -n > example.txt

Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from operation "Creating an empty file"

Ironically article calls "This is an easy one"

Re: How to write idempotent Bash scripts

#22

Post by pas » Sun, Jul 07, 2019, 3:35 PM UTC

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is.

Usually those cookbooks will want to have things in a particular way, which might not be the way you want.

Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing.

This makes them slow, hard to debug, hard to read and even write (in my opinion).

Sure, bash is hell, but the go from bash to something like Rust, instead of Chef/Ansible.

Sure, again, this might sound rather unconventional, but you get safety, speed and convenience of a full fledged programming language, single binary output, etc. And you need to test cookbooks/playbooks too, so why spend enormous effort on cobbling together scripts and high-level abstractions instead of writing just what you need with the assumptions you really have - instead of what a general cookbook/playbook has to have (which is a vast difference).

Re: How to write idempotent Bash scripts

#23

Post by zimbatm » Sun, Jul 07, 2019, 3:42 PM UTC

aey wrote: post #3

Why wouldn’t you use make? Side-effect tracking and cleanup is the job of a build tool.

make only knows how to track side-effects on files though.

A better system needs to be able to track directories creation, package installation, filesystems creations and mounting, service restarts and so on.

Re: How to write idempotent Bash scripts

#24

Post by zamadatix » Sun, Jul 07, 2019, 3:42 PM UTC

"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed."

Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?

Re: How to write idempotent Bash scripts

#25

Post by zimbatm » Sun, Jul 07, 2019, 3:45 PM UTC

All the examples are more-or-less idempotent. For example the `blkid /dev/sda1 || mkfs.ext4 /dev/sda1` doesn't guarantee what type of filesystem the partition will have. `touch file` changes the mtime of the file. `ln -sf` will fail if the target is a directory and is not atomic.

It would be nice to have a toolset of commands that are all idempotent to make that type of task easier.

Re: How to write idempotent Bash scripts

#26

Post by prepend » Sun, Jul 07, 2019, 3:51 PM UTC

cjvirtucio wrote: post #19

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

If possible, I’ll try to make a symlink just point to the staging dir after tests complete. This isn’t always possible, but makes it easy to roll back to the old prod folder if anything goes awry.

Re: How to write idempotent Bash scripts

#27

Post by [deleted] » Sun, Jul 07, 2019, 3:52 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

[deleted]

Re: How to write idempotent Bash scripts

#28

Post by joshspankit » Sun, Jul 07, 2019, 3:54 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

In other words: When promiscuous touching, always make sure you’re not making things unsanitary.

Re: How to write idempotent Bash scripts

#29

Post by grzm » Sun, Jul 07, 2019, 3:55 PM UTC

zamadatix wrote: post #24

"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed." Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?

Idempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.

Re: How to write idempotent Bash scripts

#30

Post by empath75 » Sun, Jul 07, 2019, 3:56 PM UTC

pas wrote: post #22

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

Writing a shell script in rust would be obnoxiously difficult for no real benefit.