Another tricky thing: recursively copying a folder is not easy to get right in an idempotent way due to arcane bash behaviors:
https://unix.stackexchange.com/questions/228597/how-to-copy-...
How to write idempotent Bash scripts
141–150 of 195 posts
Re: How to write idempotent Bash scripts
#142Earlier quoted context omitted.
Just have a database table, and update it with a special key. When the transaction commits, you know the script has executed correctly, and the key is written as a result. When the script doesn't complete, the commit is not performed, and you can run the script again. No need for "idempotent" scripts. Trying to do this with a filesystem instead of a real database, and things get messy.
Without proper locks, this sets you up for race conditions.
Any modern database will perform the locking for you as part of running code inside a transaction.
Re: How to write idempotent Bash scripts
#143Earlier quoted context omitted.
You still have to make that messy choice between at-most-once and at-least-once execution of each transactionally guarded section.
The idea is that you run the entire script as a single transaction. Then you only have to check for a key once (which is trivial).
Seems risky. What if it blows up, or is killed, halfway through?
I'm into using transactions as explicit guards around chunks of known-non-reversible logic as a robust way to warn that manual intervention may be necessary, but using them to replace lockfiles seems to have the worst drawbacks of both systems: it's inherently at-least-once, plus it's substantially more risk surface than a lockfile to establish and maintain a database connection/transaction.
Other concerns:
What if the database has gone away or crashed by the end of your possibly-long-running script (unless you're using sqlite, in which case, why not use a lockfile? On second thought, sqlite is probably better at lockfiles than flock...)?
What about the tech choices that using a transaction precludes? You can't write ops scripts in pure shell any more (without some seriously weird file descriptor munging and job backgrounding to keep the transaction open while you do stuff). Installing your runtime of choice plus a database driver may be unnecessary and time/space-consuming on a lot of systems you need to manage this way.
You now also have a bootstrap problem: your database driver and associated tooling may not already be available on your target system, so using such transaction-managed scripts to provision clean/empty systems is another area where additional challenges emerge.
All of those can be mitigated or worked around, and this shouldn't be taken as advice to never use an RDBMS's transaction system to manage ops tasks, but I think the cases where it adds more than it costs are pretty rare.
Re: How to write idempotent Bash scripts
#144Re: How to write idempotent Bash scripts
#145Earlier quoted context omitted.
I've always heard "deterministic" or "reproducible," curious if the term "hermetic" is somehow different in this context.
Google’s release engineering docs describe hermetic build processes as those which are “insensitive to the libraries and other software installed on the build machine. Instead, builds depend on known versions of build tools, such as compilers, and dependencies, such as libraries. The build process is self-contained and must not rely on services that are external to the build environment.” Basically, if your build pro…
I agree that deterministic would've been a better choice in this context.
I work on things close to Bazel, and the word "hermetic" gets thrown around a lot. And because of that, hermetic in my mind gets translated to "how an ideal build of a project should behave" (which obviously is wrong).
Re: How to write idempotent Bash scripts
#146A good way to remember idempotent is to think of the 'delete' or 'mark as read' function for gmail. You can delete an email or mark it as read even if it's been done before (on another open screen as an example).
Actually I think the "summon lift" button is a better example of idempotence. It doesn't matter how many times you mash the button, the lift is coming ASAP, no quicker. Hitting the button when it's lit doesn't cancel summoning the lift.
By 'lift' you mean 'elevator'? When I first read this I thought you meant 'summon lyft' the ride service but then realized what you meant!
Interestingly in the 'old days' of elevator operators hitting it multiple times would be either a 'hurry up' for the operator or an annoyance that maybe made it come slower!
Re: How to write idempotent Bash scripts
#147Earlier quoted context omitted.
Better examples: Write("filename", position, data) is idempotent; Append(handle, data) is not.
Write("filename", position, data) inputs 3 arguments and outputs probably only 1, a file. So is not a function that you can convolute, you cannot pass the output of Write() as the input to another Write(). So instead, f: SetFirstWordOfFileToZeros(file)=>file , would be a better example if we want file functions. Then SetFirstWordOfFileToZeros(SetFirstWordOfFileToZeros(file))=SetFirstWordOfFileToZeros(file) .
Yes mathematically composability is part of it. That's because math functions return the same type. Its easy to express the 'state' of a function as the value of the operation.
There's a wider idea of idempotency in computing - that state changes in general respond to operations. Repeating the same function more than once (because the network stuttered or the sender resent the message) and arriving at the same state is a common example.
Re: How to write idempotent Bash scripts
#148This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/
That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.
>The article is about writing bash scripts that don't care if you've ran them once, or 10 times.
Is the idea that participants don't know what "idempotent" means in a thread with that word in the title?
Re: How to write idempotent Bash scripts
#149Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp
Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/
You can invoke mktemp(1) in ways that work across most contemporary systems. At least if you don't care that the resulting names are going to look a bit weird (have lots of XXXXXX in them due to name prefix vs. template differences among implementations).
Re: How to write idempotent Bash scripts
#150I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.
I agree, in general, but I do think it is worth pointing out that bash is very good at what it was made for---i.e., starting processes, redirecting their output, etc.
Compare:
{ a; b && c; } ≫ log.txt
from subprocess import check_call, CalledProcessError
f = open("log.txt", "a+")
try:
check_call(["a"], stdout=f)
except CalledProcessError as e:
pass
try:
check_call(["b"], stdout=f)
check_call(["c"], stdout=f)
except CalledProcessError as e:
exit(e.returncode)
exit(0)