Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

71–80 of 195 posts

Re: How to write idempotent Bash scripts

#71
post #32

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp

Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/

Re: How to write idempotent Bash scripts

#72
post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

Python startup time is just not in the same league as bash though, if you tried to replace bash across the board you'd end up with lots of unnecessary lag.

Is that really true with limited imports though, and/or with precompiled bytecode?

I was surprised to find that Ubuntu’s handler for unknown commands that tries to suggest what you could apt install is written in Python. I’ve never noticed it being slow.

Re: How to write idempotent Bash scripts

#74
post #65

Earlier quoted context omitted.

If a file has a line ending in it, is it truly empty?

The -n flag keeps echo from emitting a line ending.

doh! I read it as \n, but then that's for other languages, so i was just all sorts of not paying attention

Re: How to write idempotent Bash scripts

#75
post #32

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp

Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/

And anyway you really want mkstemp ;)

Re: How to write idempotent Bash scripts

#76
post #38

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Op here. Exactly this. I’ve seen many times where people had scripts and had to modify it (such as commenting code) when something has failed in the middle of the script.

Re: How to write idempotent Bash scripts

#78
post #29

"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed." Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?

Idempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.

This is incorrect. Idempotency is about not needing to worry about running something more than once. If you introduce race conditions with follow-on runs, you're not idempotent.

Re: How to write idempotent Bash scripts

#79

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Op here. Yeah you’re right. I’ve mentioned it though in the context of file contents. Just updated to make it clear.

Re: How to write idempotent Bash scripts

#80
post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

Just have a database table, and update it with a special key. When the transaction commits, you know the script has executed correctly, and the key is written as a result. When the script doesn't complete, the commit is not performed, and you can run the script again. No need for "idempotent" scripts.

Trying to do this with a filesystem instead of a real database, and things get messy.

Post reply on HN