How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#91

Post by dymk » Sun, Jul 07, 2019, 6:29 PM UTC

heavenlyblue wrote: post #86

dymk wrote: post #53

Earlier quoted context omitted.

I feel like my comment went in one eyeball and out the other. Chef and friends are blocked on disk I/O. How does a typesystem and/or thinner abstraction layer to disk I/O speed up the underlying expensive operation: blocking disk I/O?

I - on the other hand, feel like your comment somehow forgot that the op said “safety, speed and convenience”, so you attacked the least important point made in the first place. It’s much easier to see the failure conditions in a Rust program rather than in bash. Also rust seems like easier to maintain, too.

How does having a borrow checker and snazzy memory safety benefit you when nearly all the operations you're performing are disk I/O?

How is a 100 lines of rust easier to maintain than 10 lines of shell script?

It's exactly what the other commnet said: "Writing a shell script in rust would be obnoxiously difficult for no real benefit."

Re: How to write idempotent Bash scripts

#92

Post by ddebernardy » Sun, Jul 07, 2019, 6:30 PM UTC

pdkl95 wrote: post #90

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

> traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state Traps are underused. They solve a lot of error-handling and cleanup problems really easily! Here[1] is a simple example of using a trap to cleanup a tempfile, that safely handles pipeline errors and unexpected exit/return. The same basic idea works for many kinds of error handling, cleanup, or similar work. If you ever ne…

Intriguing. I always trap before setting up, on the basis that a script could get interrupted before setup is completed. Is there any general agreement on what the correct order should be?

Re: How to write idempotent Bash scripts

#93

Post by iforgotpassword » Sun, Jul 07, 2019, 6:32 PM UTC

empath75 wrote: post #36

This is surely a good collection of flags to know, but I sort of object to the premise of the article. There is a good reason that many of these flags are not default behavior, and it’s that they can be quite destructive. If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite…

> If you’re writing a script, as the beginning scenario says, that has errors in it, and you expect to have more, do you really want to tell all the commands to power through and delete and overwrite stuff if it encounters an unexpected state?

Where are they saying this? These flags are about making sure the script wouldn't error out right away if you run it again, eg how a regular mkdir for a path that already exists would not exit 0 and thus end script execution (given set -e is active, which the article seems to imply). Also it is not so much about purposely writing a script that contains errors, because that would just mean the error is encountered every time, but more about transient errors like running out of disk space, a curl call failing because of network hiccups etc. Everything your script did up until that failure shouldn't prevent a second call from succeeding.

Re: How to write idempotent Bash scripts

#94

Post by faho » Sun, Jul 07, 2019, 6:34 PM UTC

achamayou wrote: post #75

arendtio wrote: post #71

Earlier quoted context omitted.

Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/

And anyway you really want mkstemp ;)

Not in a shell. There is no `mkstemp` command, at least on a typical GNU system. There is a `mktemp`, which may very well use the `mkstemp` function in its source.

I.e. this is about `mktemp(1)`, not `mktemp(3)`.

Re: How to write idempotent Bash scripts

#95

Post by grzm » Sun, Jul 07, 2019, 6:36 PM UTC

Godel_unicode wrote: post #78

grzm wrote: post #29

Earlier quoted context omitted.

Idempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.

This is incorrect. Idempotency is about not needing to worry about running something more than once. If you introduce race conditions with follow-on runs, you're not idempotent.

If race conditions are an issue within the context of your intent, then yes, you'll need to take that into account. Again, one needs to take into account the intent and context of the script. If other aspects of the environment ensure that the operations are serialized (say you're making an update to a system which you've rotated out of production for the update), you won't need to worry about race conditions. Need to make sure you have no timing effects? That will effect how the script is written. Need to worry about CPU or IO utilization (including reads)? That will need to be considered.

Re: How to write idempotent Bash scripts

#96

Post by rwestergren » Sun, Jul 07, 2019, 6:38 PM UTC

Context and purpose of the bash script in question is important here. In the example, the author is writing a simple bootstrap script for a dev machine. A number of the critiques here, while valid, are aimed at different use-cases.

Re: How to write idempotent Bash scripts

#97

Post by antpls » Sun, Jul 07, 2019, 6:38 PM UTC

pletnes wrote: post #83

rb808 wrote: post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

If you’re into minimal docker images, bash or similar shell might be the only script interpreter available. Python might not exist on your windows machine. Etc.

Surely we could all agree on a minimal subset of useful Python to write scripts : loops, arrays, lists, some strings functions, etc. The resulting Python interpreter would probably be tiny.

For example, there is https://docs.bazel.build/versions/master/skylark/language.ht... which is a subset of Python.

Re: How to write idempotent Bash scripts

#98

Post by riazrizvi » Sun, Jul 07, 2019, 6:45 PM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

I've heard Idempotency confused with Consistency. Idempotency is where a function f can be applied (as a composition) multiple times and it gives the same result, so Idempotency: f(f(x)) = f(x). Whereas Consistency: f(x) = f(x) = f(x). An example of an idempotent function f is RaiseToThePowerZero(x) on x > 0. We usually take consistency for granted, so it's perhaps better to think of an example of a non-consistent function f which would be RandomNumberGenerator().

Re: How to write idempotent Bash scripts

#99

Post by gabrielblack » Sun, Jul 07, 2019, 6:47 PM UTC

zokier wrote: post #21

The first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be echo -n > example.txt Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from…

"echo -n " is useless, is sufficient:

> example.txt

without prepending the echo command. This will create an empty file or it will truncate an existent file. The touch command could be preferable to preserve the content, if needed.

Re: How to write idempotent Bash scripts

#100

Post by darkerside » Sun, Jul 07, 2019, 6:47 PM UTC

RijilV wrote: post #87

kousikk wrote: post #35

Earlier quoted context omitted.

Last modified timestamps in source code are a bad idea and they make builds non-hermetic. Better to fix up those scenarios in code than work around them. Irrespective you have a good point that touch does have side effects.

I hate to nitpick, but from the man page of touch: > touch -- change file access and modification times If one were to argue side effects of touch, it would be that non-existent files are created. The purpose of touch is to update access and modification times.

I think you're misunderstanding the way the term side effect is being used in this context. A side effect can refer to any impacts on a system that happen outside of the scope of the code itself. In this case, the primary intended effect of "touch" is to initiate the side effect of modifying file access and modification times.

https://en.m.wikipedia.org/wiki/Side_effect_(computer_scienc...