How to write idempotent Bash scripts • Hacker News Board

Re: How to write idempotent Bash scripts

#161

Post by iteriteratedone » Mon, Jul 08, 2019, 2:06 AM UTC

Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

The fact is they shouldnt have to.

The env should be known. If you cant control the env then you shouldnt be writing the script. If someone runs your script on a windows machine and it fails its not the script writers fault. There is a level of assumed resources.

Re: How to write idempotent Bash scripts

#162

Post by manigandham » Mon, Jul 08, 2019, 2:10 AM UTC

dymk wrote: post #44

empath75 wrote: post #30

Earlier quoted context omitted.

Writing a shell script in rust would be obnoxiously difficult for no real benefit.

Yeah, and tools like Chef are 99% blocked on disk operations anyways, so I have no idea why GP thinks Rust would help

It's about functionality and maintainability, not performance. If you have complicated scripts then they start to resemble actual programs, at which point you might be better off using a real programming language to code.

I wouldn't pick Rust for this but there are lots of examples of using Node/JS, or C# or Python. It's a much nicer environment and benefits from full IDE support.

For a good example, look at what Pulumi is doing for modern infrastructure: https://www.pulumi.com/

Re: How to write idempotent Bash scripts

#163

Post by ggm » Mon, Jul 08, 2019, 2:45 AM UTC

grzm wrote: post #29

zamadatix wrote: post #24

"The -f flag removes the target destination before creating the symbolic link, hence it’ll always succeed." Removing something and adding it back seems to be, by definition, not idempotent. What if something tries to access that file in between? What about the filesystem timestamps (same issues as with the "touch" claim)?

Idempotency focuses on the result, rather than implementation. If the intent of the script is to ensure the file is there when the script is finished, then it’s idempotent. If the time stamps matter for the intent of the script, then, yes, what you pointed out would be an issue.

The chmod command can alter the flagset on the link not the target, if you select the right options.

How do you guarantee the replacement symlink had the same chmod flags and grp flags as the original? (chmod -h for reference*)

I don't think these commands are idempotent in the wide. They satisfy only the narrowest "yea, most normal-ish cases are sort of maybe ok" but since we lost atomicity, there are significant windows of time in the filesystem where inodes are in flux, and the new thing has a different inode to the old thing, if you delete and replace it. The disk is different.

Re: How to write idempotent Bash scripts

#164

Post by da_chicken » Mon, Jul 08, 2019, 3:30 AM UTC

bdcravens wrote: post #9

da_chicken wrote: post #7

Earlier quoted context omitted.

Why would you use a build tool when you're not writing a build script?

It's not necessarily a build tool - it's commonly used for that, but it's a useful DSL. rake in the Ruby world is used for all sorts of tasks for web apps and more, even though it was originally designed as the Ruby version of make. Look at npm - the NODE package manager, but it quickly grew to be used for more than server-side applications. A title is a description, not a constraint.

Just because a language can do the task doesn't mean it's actually a good idea to use it for that task.

Make is much less well known than shell script. In fact, I'd go so far as to say that make is infamously poorly understood and makefiles are infamously poorly written. That makes it less maintainable, or otherwise you'll need to put "experienced with GNU make" on your job description.

If you've got the script scheduled, then you're going to have to document it everywhere that it's not actually a build script so please don't disable it because it doesn't obviously look like it needs to be running every 6 hours.

And because it's a DSL and you're somewhat going out of scope, you're more likely to have a problem like needed to extend your script to do something you can't do as easily in make. Worse, you might want to do something that you're expressly not supposed to do [0].

You're going to have to defend your decision every time you present the script, too. And if you ever need to pass it off, the first thing you're going to have to do is explain why you used make. Whether or not the person you hand it of too is an idiot or not, you know the first question is going to be "why isn't this a shell script or Python script?" You're going to have to defend your decision to use make instead of shell script because of idempotence, even though you can write idempotent bash scripts. And if the person you're talking to is an idiot -- and let's be fair that there's a good chance that that is the case -- then they'll never figure it out.

So, for me, you've got to go beyond "this language can technically do the task" in order for me to understand why you'd want to use something that's generally understood to be used for build scripts for, well, anything other than that. By choosing make, you're doing something unexpected. That's a bad idea.

[0]: https://www.gnu.org/prep/standards/html_node/Utilities-in-Ma...

Re: How to write idempotent Bash scripts

#165

Post by mehrdadn » Mon, Jul 08, 2019, 3:47 AM UTC

NelsonMinar wrote: post #13

I got hung up on the claim "Touch is by default idempotent. ... A second call won’t have any effects". The whole point of touch is to have an effect every single time you run it. It updates the file atime and mtime. That may seem harmless in your application, but it's definitely not no effect. Also promiscuous touching is the source of a bunch of bogus last modified dates in source code bundles.

Just a note... even reading a file updates the access time, so if you reason this way then no program touching any files in the worktree will ever be idempotent... which seems to lose the utility of the concept.

Re: How to write idempotent Bash scripts

#166

Post by unixhero » Mon, Jul 08, 2019, 5:21 AM UTC

news_hacker wrote: post #107

dymk wrote: post #38

Earlier quoted context omitted.

That's orthogonal to the article - in fact, the article hints that you'd take this advice when your bash script is failing midway, such as it might with `set -e`. The article is about writing bash scripts that don't care if you've ran them once, or 10 times.

Seems like all the pedants came out for this article. In good-faith I understand the principles OP is suggesting and have first-hand experience of their usefulness. But it's easier to nit-pick I guess.

Everytime I discuss Bash on the net, these gremlin style characters ruin the fruitful discussions I've had ... I think the article provides excellent advice.

Re: How to write idempotent Bash scripts

#167

Post by mappu » Mon, Jul 08, 2019, 7:32 AM UTC

fwip wrote: post #65

dylan604 wrote: post #51

Earlier quoted context omitted.

If a file has a line ending in it, is it truly empty?

The -n flag keeps echo from emitting a line ending.

I used a distro in recent memory where `echo -n foo` really produced `-n foo`.

The workaround was `printf "%s" foo`.

Re: How to write idempotent Bash scripts

#168

Post by ptman » Mon, Jul 08, 2019, 7:35 AM UTC

pas wrote: post #119

lukeschlather wrote: post #46

Earlier quoted context omitted.

Rust doesn't get you anything resembling what you want here. Chef/Ansible/Salt could be described as rudimentary type systems for operating system states, that enable you to convert between different types/states. It's conceivable you could build such a system in Rust, but Rust itself has no primitives that would make it easy. It certainly isn't a good starting point when you want to set up a database server.

All Config Management systems execute other programs, parse their output, check/parse files underneath their fancy skin. I've used both Chef and Ansible. Their maintenance cost is pretty high unfortunately, and they are not that flexible, nor resilient to worth it. No wonder the container/k8s boom is so big. Immutable infrastructure (docker images) with really powerful operational features is what can justify high ma…

Nix/guix might be a better approach. Ansible and chef describe part of the desired state of a system and require a very large definition to cover the whole system. nix aims to track the whole system state and allow you to supply changes which again combine to another whole system state

Re: How to write idempotent Bash scripts

#169

Post by gabrielblack » Mon, Jul 08, 2019, 7:39 AM UTC

gabrielblack wrote: post #99

zokier wrote: post #21

The first example ("idiom") is bit tricky, because the title is "Creating an empty file". An alternative would for example be echo -n > example.txt Both fit the description of "Creating an empty file". Is it idempotent, or more or less so than `touch example.txt`? If there is already an existing file, then touching it obviously will not empty it, so the end state is not "empty file exists" like one would expect from…

"echo -n " is useless, is sufficient: > example.txt without prepending the echo command. This will create an empty file or it will truncate an existent file. The touch command could be preferable to preserve the content, if needed.

P.S.

>> example.txt

this one (with a double > ) could replace touch in this application case: it's less readable but more efficient.

Re: How to write idempotent Bash scripts

#170

Post by posix_me_less » Mon, Jul 08, 2019, 9:05 AM UTC

arendtio wrote: post #71

bonyt wrote: post #32

Don’t forget mktmp for making temporary files or directories, instead of putting files in a set place in /tmp. https://linux.die.net/man/1/mktemp

Well, sadly mktemp is not part of POSIX. So while it is available on a wide range of systems, there are also different implementations with different options. So while I like the job it does, I am also pretty disappointed how 'unportable' scripts become as soon as they use mktemp :-/

It is sad to read these comments by people believing POSIX is something relevant today. My advice is to give up on false hope that POSIX will solve portability problems. POSIX ideas of shell are 30 years old and refusing to use anything newer alone won't make your script work correctly on all systems. If feasible, install bash, which has become the standard de facto. Or just write your script to test for the shell version and then launch appropriate code path.