Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

Post reply on HN
Print view View original

101–110 of 195 posts

Re: How to write idempotent Bash scripts

#101

Post by JoeAltmaier »

riazrizvi wrote: post #98
NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

I've heard Idempotency confused with Consistency. Idempotency is where a function f can be applied (as a composition) multiple times and it gives the same result, so Idempotency: f(f(x)) = f(x) . Whereas Consistency: f(x) = f(x) = f(x) . An example of an idempotent function f is RaiseToThePowerZero(x) on x > 0. We usually take consistency for granted, so it's perhaps better to think of an example of a non-consistent…

Better examples: Write("filename", position, data) is idempotent; Append(handle, data) is not.

Re: How to write idempotent Bash scripts

#102

Post by suprgeek »

A lot of the suggestions in this post are "idempotent" only in the sense of the specific use case that the author is interested in. Please do not take this advise to be generally applicable for the uses of "idempotent" or in general to mean best practice while writing bash scripts.

The advice such as replace rm with rm -f could lead to Disaster depending on the scenario. So a HUGE YMMV

Re: How to write idempotent Bash scripts

#103

Post by xorcist »

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Absolutely this. When the intent is to get the system into a desired state, reach for Puppet/Salt/Ansible.

If you are doing sysop^Wdevops work it is the single most important thing you can learn, once you have a good grasp of your shell and your editor.

The difference between a giant git archive of shell scripts that various people have modified over the years and state changes described in a configuation management language is the difference between fixing things in the small and reasoning about integrated systems. It's something that needs to be experienced to be appreciated.

Reasoning about system state as an integrated whole is just as relevant when you are shipping applications as containers, if not more so. It's not uncommon to start using something like Kubernetes without first being able to describe global state and the result is just as messy as before, if not worse. Something like Helm is impossible to understand unless you have complete control over your configuration.

Re: How to write idempotent Bash scripts

#104

Post by pdkl95 »

ddebernardy wrote: post #92
pdkl95 wrote: post #90

Earlier quoted context omitted.

> traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state Traps are underused. They solve a lot of error-handling and cleanup problems really easily! Here[1] is a simple example of using a trap to cleanup a tempfile, that safely handles pipeline errors and unexpected exit/return. The same basic idea works for many kinds of error handling, cleanup, or similar work. If you ever ne…

Intriguing. I always trap before setting up, on the basis that a script could get interrupted before setup is completed. Is there any general agreement on what the correct order should be?

It probably depends on if the cleanup is safe to run without its setup. For "rm -f", that's true so the trap probably should be first in that case. If you have to worry about strict dependency ordering, trapping first may not be possible.

Of course, the better solution to those situations is to make the cleanup command idempotent ...

(that said, any order is much better than the unfortunately-common style of simply ignoring errors and exceptions.)

Re: How to write idempotent Bash scripts

#105

Post by riazrizvi »

JoeAltmaier wrote: post #101
riazrizvi wrote: post #98

Earlier quoted context omitted.

I've heard Idempotency confused with Consistency. Idempotency is where a function f can be applied (as a composition) multiple times and it gives the same result, so Idempotency: f(f(x)) = f(x) . Whereas Consistency: f(x) = f(x) = f(x) . An example of an idempotent function f is RaiseToThePowerZero(x) on x > 0. We usually take consistency for granted, so it's perhaps better to think of an example of a non-consistent…

Better examples: Write("filename", position, data) is idempotent; Append(handle, data) is not.

Write("filename", position, data) inputs 3 arguments and outputs probably only 1, a file. So is not a function that you can convolute, you cannot pass the output of Write() as the input to another Write(). So instead, f: SetFirstWordOfFileToZeros(file)=>file, would be a better example if we want file functions. Then SetFirstWordOfFileToZeros(SetFirstWordOfFileToZeros(file))=SetFirstWordOfFileToZeros(file).

Re: How to write idempotent Bash scripts

#107

Post by news_hacker »

dymk wrote: post #38
united893 wrote: post #33

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Seems like all the pedants came out for this article. In good-faith I understand the principles OP is suggesting and have first-hand experience of their usefulness. But it's easier to nit-pick I guess.

Re: How to write idempotent Bash scripts

#108

Post by gen220 »

rectangletangle wrote: post #85
pmarreck wrote: post #58

Earlier quoted context omitted.

I prefer "deterministic" in this case (I think?)

I've always heard "deterministic" or "reproducible," curious if the term "hermetic" is somehow different in this context.

Google’s release engineering docs describe hermetic build processes as those which are “insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment.”

Basically, if your build process requires you to pull code from any repository that you do not own, it isn’t hermetic. If you have to `pip install` or `go get` a third party (or even in-house!) dependency from a source that you do not control, your build is not hermetic.

Effectively, this means that you have to have versioned copies of all of your third party dependencies, and version-specified build graphs. Very hard to do without a mono repo and a build tool.

In the context of the GP, I’d say that deterministic would have been a better word choice. A reliance on time stamps technically wouldn’t make a build process non-hermetic, but it would definitely make it non deterministic. It’s technically possible to have hermetic builds without having reproducible builds, although that would be a very bizarre org. :)

Re: How to write idempotent Bash scripts

#109

Post by gabrielblack »

bonyt wrote: post #32

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp

mktemp couldn't make a script idempotent. I remember a script that failed because it needed to create temporary executables in /tmp. To apply some hardening policies, someone remounted /tmp in "noexec" mode and ... kaboom. So if you use that, better if you specify TMPDIR with proper checking.
Post reply on HN