Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

61–70 of 195 posts

Re: How to write idempotent Bash scripts

#61
post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

Python startup time is just not in the same league as bash though, if you tried to replace bash across the board you'd end up with lots of unnecessary lag.

Re: How to write idempotent Bash scripts

#62
post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

You might find rsync cleaner to use than a diff/cp loop.

Another approach is to create a unique directory (named by hash or date) with the results, and once it's good, symlink the real directory to it. (Full disclosure, I stole this approach from nextflow.io, a great workflow system aimed at bioinformatics).

    ln -sfn "build-cache/2019-07-07-21:23:04" "build"

Re: How to write idempotent Bash scripts

#63

Earlier quoted context omitted.

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

What do you use to compute the diff? Just the diff command itself on individual files?

Could use something like rsync to automatically check for a difference and copy

Re: How to write idempotent Bash scripts

#64
post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

Assuming umask is one that bites any dev that has never worked with hardened systems.

Re: How to write idempotent Bash scripts

#65
post #21

The first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be echo -n > example.txt Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from…

If a file has a line ending in it, is it truly empty?

The -n flag keeps echo from emitting a line ending.

Re: How to write idempotent Bash scripts

#66
post #49

A good way to remember idempotent is to think of the 'delete' or 'mark as read' function for gmail. You can delete an email or mark it as read even if it's been done before (on another open screen as an example).

Actually I think the "summon lift" button is a better example of idempotence. It doesn't matter how many times you mash the button, the lift is coming ASAP, no quicker. Hitting the button when it's lit doesn't cancel summoning the lift.

Re: How to write idempotent Bash scripts

#68
post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

While not disagreeing, I would not agree that "everyone should go to Python" since I would personally prefer (for example) Ruby or Elixir, which can work just as well as scripting languages and are at least as readable, while also not triggering too much "VM startup" time.

Of course, others would prefer, say, Clojure, or Kotlin (although the JVM startup time is the worst of all), but therein lies the issue... We're stuck with Bash scripting because everyone (more or less) knows/understands it to some extent, it's installed everywhere, and it has essentially no startup-time cost.

Re: How to write idempotent Bash scripts

#69
post #43

Great article, but a common scenario is missing: I want to run a long running script, but not if another copy of it is already running.

https://linux.die.net/man/1/flock would probably help

A word of caution on flock: it may fail when used on some network filesystems, and it is tricky to make sure you've got it right. If you're locking on-disk it's pretty straight-forward, though.

Re: How to write idempotent Bash scripts

#70
post #42

Let's try to avoid using a hammer on anything that resembles a nail: https://github.com/valvesoftware/steam-for-linux/issues/3671

How is that related to the article aside from "both involve bash"?

It's because rm -rf "$STEAMROOT/"* will evaluate as rm -rf "/"* if $STEAMROOT for some reason is blank or unset, which is what happened here. (As someone in that discussion mentioned, a little more Bash knowledge might have suggested using rm -rf "${STEAMROOT:?}/"* instead, to force it to (at least) error if it is empty or unset.)
Post reply on HN