How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#121

Post by dredmorbius » Sun, Jul 07, 2019, 7:45 PM UTC

kahlonel wrote: post #115

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

> An easy way to cover that is to have the script create a file, which you check at the start of the script and if present you exit. This does sound a good solution on paper, but "checking at the start" and "creating a file" are two different steps (i.e. not atomic) and will cause trouble eventually if your system has the tendency to run the script twice. A better solution is to use the `flock` command before creatin…

Within a shell script context, 'mktemp' and 'mv' are almost always what you want, possibly 'ln -s' for libraries.

The first safely creates a temporary file. The second (with noclobber set or -n argument) will atomically rename a file, but not overwrite existing contents.

Linux uses symlinks to system library files in order to allow in-place atomic replavement without affecting running processes with open filehandles to earlier versions. MS Window's lack of this feature is what makes (or made, I'm very out of date) malware removal and system updates so painful.

Re: How to write idempotent Bash scripts

#122

Post by dredmorbius » Sun, Jul 07, 2019, 7:48 PM UTC

nerdponx wrote: post #54

cjvirtucio wrote: post #19

Earlier quoted context omitted.

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

What do you use to compute the diff? Just the diff command itself on individual files?

diff -R

You can use numerous tools to compare/manage branches; md5sum / shasum (md5 is usually useful, though not entirely safe), diff and kin, vimdiff, rsync, fdupes, jdupes, git, hg.

Re: How to write idempotent Bash scripts

#123

Post by joshuamorton » Sun, Jul 07, 2019, 7:49 PM UTC

gen220 wrote: post #108

rectangletangle wrote: post #85

Earlier quoted context omitted.

I've always heard "deterministic" or "reproducible," curious if the term "hermetic" is somehow different in this context.

Google’s release engineering docs describe hermetic build processes as those which are “insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment.” Basically, if your build pro…

One way of putting this might be that deterministic implies that the same input artifacts give the same output artifacts on the same machine.

Hermetic implies that unexpectedly different input artifacts are not possible, and builds are deterministic across machines/environments.

Re: How to write idempotent Bash scripts

#124

Post by dredmorbius » Sun, Jul 07, 2019, 7:49 PM UTC

amelius wrote: post #80

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

Just have a database table, and update it with a special key. When the transaction commits, you know the script has executed correctly, and the key is written as a result. When the script doesn't complete, the commit is not performed, and you can run the script again. No need for "idempotent" scripts. Trying to do this with a filesystem instead of a real database, and things get messy.

Without proper locks, this sets you up for race conditions.

Re: How to write idempotent Bash scripts

#125

Post by dredmorbius » Sun, Jul 07, 2019, 7:50 PM UTC

news_hacker wrote: post #107

dymk wrote: post #38

Earlier quoted context omitted.

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Seems like all the pedants came out for this article. In good-faith I understand the principles OP is suggesting and have first-hand experience of their usefulness. But it's easier to nit-pick I guess.

This is a case where the pedents should at least be considered.

Re: How to write idempotent Bash scripts

#126

Post by pas » Sun, Jul 07, 2019, 7:51 PM UTC

mnutt wrote: post #50

pas wrote: post #22

Earlier quoted context omitted.

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

Yeah, I find out-of-the-box cookbooks about as helpful as out-of-the-box docker images: useful as a starting place but to leverage chef you really need to write your own. Chef resources/providers are nicely composable and the primitives are pretty easy to rewrite if you want, my main point is that chef gets you to think in terms of convergence instead of brittle imperative changes. You could rewrite your bash scripts…

My comment tried to argue that Chef's convergence is nothing more than a well hidden imperative thing.

I agree that manipulating existing machines is folly, but that's why people use containers, and immutable images. And then you can prepare the images anyway you like, but since they are very deterministic (modulo the external repositories/packages/curl-bash-piped-scripts), you usually use something simple (eg bash), because there's no need for that state management.

And actual state management should be left for the platform (k8s, swarm, or some cluster manager).

I found the chef workflow slow, not just the executor itself. (Uploading cookbooks, fiddling with dependencies, testing them, debugging them, etc.)

Ansible was even worse in my experience. Much slower execution, harder to debug, opaque python blobs, extremely confusing and fragile "YAML programming" coupled with hostile variables files. (All the things that are tenfold more intuitive in Chef.)

Re: How to write idempotent Bash scripts

#127

Post by vonseel » Sun, Jul 07, 2019, 7:52 PM UTC

Sounds like a bunch of half-baked advice from a mid-level developer who knows a little bit but thinks he knows a lot. As others have mentioned, some of this advice is rather dangerous or only appropriate in particular use-cases.

Call me an old bastard, but I'm always suspicious people just write blogs like this to make themselves look like an expert and throw another professional activity on their resumes.

Re: How to write idempotent Bash scripts

#128

Post by argd678 » Sun, Jul 07, 2019, 7:55 PM UTC

iforgotpassword wrote: post #10

argd678 wrote: post #2

The problem with the idempotent trend in configuration management is that it’s all based on not tracking or knowing what the current state of something is. So reasoning about how these systems work is fundamentally impossible. It would be better to focus on systems where by we can always know the state and improve the tooling there.

It's a pragmatic approach to this problem. Usually tracking state means that you write some version number or similar to a file, but than if for some reason that goes out of sync with the actual state, for example because you put it in /etc and someone accidentally restores an older version of it, hilarity ensues, because it's guaranteed the devs never expected this to happen and don't handle it in any way. For the c…

The reason it’s pragmatic is due to how systems are designed to begin with, it’s not what you want.

We know what the state is right after the command for an undetermined amount of time, but we have no knowledge of what’s changing it. Why not cover that? There’s no reason not to other than it hasn’t been worked on enough. Consider a database schema migration, there’s a reason those are not idempotent, you know each state. Seems quite solvable.

Re: How to write idempotent Bash scripts

#129

Post by ithkuil » Sun, Jul 07, 2019, 8:03 PM UTC

rectangletangle wrote: post #85

pmarreck wrote: post #58

Earlier quoted context omitted.

I prefer "deterministic" in this case (I think?)

I've always heard "deterministic" or "reproducible," curious if the term "hermetic" is somehow different in this context.

The (subtle?) distinction is around "what it is" rather than "what it is for":

A deterministic process always produces the same output given the same set of inputs.

A hermetic behavior ensures that indeed you'll always have the same inputs. It can just be a set of best practices (e.g. being very careful of not depending on external inputs that might change outside of your control) or it can involve an active barrier that sandboxes your environment in order to ensure that you indeed always have the same inputs.

A reproducible process is a process that can be repeated later in the future. There are various degrees of reproducibility you might be interested in. For example, you might want "bit for bit" reproducibility (important for security) or you just want to make sure you can rebuild something functionally equivalent (e.g. the compilation or link phase might not be fully deterministic in the order and layout of compilation units).

Reproducible processes usually rely on a deterministic system and leverage hermetic behaviours to ensure reproducibility (over time or across locations)

Re: How to write idempotent Bash scripts

#130

Post by pletnes » Sun, Jul 07, 2019, 8:12 PM UTC

antpls wrote: post #97

pletnes wrote: post #83

Earlier quoted context omitted.

If you’re into minimal docker images, bash or similar shell might be the only script interpreter available. Python might not exist on your windows machine. Etc.

Surely we could all agree on a minimal subset of useful Python to write scripts : loops, arrays, lists, some strings functions, etc. The resulting Python interpreter would probably be tiny. For example, there is https://docs.bazel.build/versions/master/skylark/language.ht... which is a subset of Python.

Micropython runs in 16k RAM.

https://micropython.org/