Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

Post reply on HN
Print view View original

181–190 of 195 posts

Re: How to write idempotent Bash scripts

#181

Post by audience_mem »

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Is it not a bit of a large jump from "shell script" to "chef"? There is some inbetween.

I understand a shell script is not suitable for orchestrating a datacenter, but sometimes it is suitable for installing a couple of things.

Re: How to write idempotent Bash scripts

#182

Post by theamk »

jrumbut wrote: post #18
intc wrote: post #14

> This is an easy one. Touch is by default idempotent. This means you can call it multiple times without any issues. A second call won’t have any effects Every call to touch will alter the modification time of the file (example.txt in this case).

It will change the access time too. Almost every suggestion in the post has this problem which is unfortunate since access/modify/create time are used in a lot of scripts to see which files were or were not processed by prior runs of a script (whether this is a good idea or not). It is very difficult to write truly idempotent bash scripts (consider log files). Creating sort of idempotent or at least rerunnable script…

Do people actually use access time for anything?

I occasionally do full-text searches on system files, and this resets access times on all files. Thus, I cannot imagine using a script which cares about it.

Re: How to write idempotent Bash scripts

#183

Post by theamk »

arendtio wrote: post #73

For everybody interested in portable shell programming I like to recommend the following page which offers a good overview over POSIX compliant commands with links directly into the POSIX standard: https://shellhaters.org

Is POSIX relevant today? I'd think that overwhelming majority of the shell scripts would require either Linux or BSD dialects.

And if one wanted to limit my shell scripts significantly, I'd limit myself to commands supported by busybox's "ash" shell -- this is a limited shell environment which is pretty widely used in initrd's and embedded devices.

Re: How to write idempotent Bash scripts

#184

Post by posix_me_less »

arendtio wrote: post #180
posix_me_less wrote: post #170

Earlier quoted context omitted.

It is sad to read these comments by people believing POSIX is something relevant today. My advice is to give up on false hope that POSIX will solve portability problems. POSIX ideas of shell are 30 years old and refusing to use anything newer alone won't make your script work correctly on all systems. If feasible, install bash, which has become the standard de facto . Or just write your script to test for the shell v…

I know that POSIX doesn't meet the expectations many of us have, but I don't see how it isn't relevant anymore? I mean, if you want to write portable shell scripts it is still a good reference on what you can expect to find on many systems (and which are options introduce by GNU and the likes). Yes, ultimately you have to test your scripts on the actual systems, but that is something you have to do anyway. For exampl…

POSIX is part of unix history and running systems have some compatibility level with it, so in that respect it is relevant. But as for writing universally working scripts, it is not a panacea, because systems shells are not exactly POSIX compliant. For example, bash and Linux, the standard of unix de facto, deviate from POSIX, the standard de iure.

> but that is something you have to do anyway.

Exactly my point.

Re: How to write idempotent Bash scripts

#185

Post by pnutjam »

audience_mem wrote: post #181
mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Is it not a bit of a large jump from "shell script" to "chef"? There is some inbetween. I understand a shell script is not suitable for orchestrating a datacenter, but sometimes it is suitable for installing a couple of things.

Ansible is good for small or large installs.

Re: How to write idempotent Bash scripts

#186

Post by pnutjam »

ptman wrote: post #168
pas wrote: post #119

Earlier quoted context omitted.

All Config Management systems execute other programs, parse their output, check/parse files underneath their fancy skin. I've used both Chef and Ansible. Their maintenance cost is pretty high unfortunately, and they are not that flexible, nor resilient to worth it. No wonder the container/k8s boom is so big. Immutable infrastructure (docker images) with really powerful operational features is what can justify high ma…

Nix/guix might be a better approach. Ansible and chef describe part of the desired state of a system and require a very large definition to cover the whole system. nix aims to track the whole system state and allow you to supply changes which again combine to another whole system state

For tracking changes:http://sysconfcollect.sourceforge.net/

Re: How to write idempotent Bash scripts

#187

Post by pas »

ptman wrote: post #168
pas wrote: post #119

Earlier quoted context omitted.

All Config Management systems execute other programs, parse their output, check/parse files underneath their fancy skin. I've used both Chef and Ansible. Their maintenance cost is pretty high unfortunately, and they are not that flexible, nor resilient to worth it. No wonder the container/k8s boom is so big. Immutable infrastructure (docker images) with really powerful operational features is what can justify high ma…

Nix/guix might be a better approach. Ansible and chef describe part of the desired state of a system and require a very large definition to cover the whole system. nix aims to track the whole system state and allow you to supply changes which again combine to another whole system state

Yes. Though I'd probably want to simply use something "trusted", either Ubuntu LTS or CentOS/RHEL, keep the installed packages to a minimum, use a local repository mirror proxy, track package changes there, etc.

And the image build should be just a simple imperative install these packages, use this config, run this command on invocation.

Nix is rather amazing with its powerful CLI stuff it provides (S3 compatible dependency store, fetching via SSH, closures, etc).

My only problem with NixOS is that it's very much like Gentoo. It has infinite composability built-in, but it means you have to rebuild everything. In Debian/Ubuntu land you usually can simply enable/disable install/uninstall specific feature related packages. (For example postfix and postfix-mysql packages.)

Re: How to write idempotent Bash scripts

#188

Post by pas »

CameronNemo wrote: post #176
pas wrote: post #137

Earlier quoted context omitted.

CM systems spend a lot of time on bootstrapping, creating their workspace, downloading the scripts, parsing them, etc. Then they simply execute other programs to then parse their output. And/or fiddle with files (parse them, alter them, write them). Sure fundamentally the syscalls and apt/dnf/yum will be the slow parts, but I found that development of CM scripts/plays/recipes are usually bottlenecked on the turnaroun…

Why do we care about speed here? Surely reliability and readability are paramount features...

Development speed is important. You can use the most reliable thing in the world if it takes a day to test/build/deploy.

And simple imperative script is a lot more readable than a custom DSL with who knows what ruby hooks.

Currently the CM system that makes sense on the long run is a git repo for Terraform. (Because everything else runs in containers anyway, and to set up the immutable images you don't need "state management". And where you need it you need active state management such as k8s and its specific operators.)

Re: How to write idempotent Bash scripts

#189

Post by pas »

eddieroger wrote: post #40
pas wrote: post #22

Earlier quoted context omitted.

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

They didn't claim Chef was magic, and yeah it's just a bunch of cookbooks, but that doesn't change their point that it's declarative and useful for describing an end-state. Chef is hard and slow if you rely solely on others' work without understanding what it's doing, which is functionally equivalent to magic. But if you know how to tune things, keep your cookbooks down to only required ones, and a few other tricks y…

I loved Chef after we got familiar with it. It was nice, cute, had solutions for a lot of problems. (foodcritic, kitchen, rubocop, etc.)

But it simply is a solution in search of a problem in today's container orchestrator/platform world.

HAproxy is amazing, but a k8s Ingress service [0] has a nice API, so I don't have to run Chef to add a new vhost. Yet I can persist the config in a YAML. (Or json, or whatever.)

[0] which can of course be backed by HAproxy down below, but traefik has dynamic config; though haproxy2 will do that too

https://www.haproxy.com/blog/haproxy-2-0-and-beyond/

Re: How to write idempotent Bash scripts

#190

Post by pas »

ajxs wrote: post #160
pas wrote: post #22

Earlier quoted context omitted.

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

Many of the comments in this thread seem to be sucked into a false dichotomy. There are better options than _either_ Rust or Bash for writing complicated shell scripts. Perl/Python are more suited to the task than either. The challenge of deciding what degree of complexity warrants the investment of more complicated language is a design decision left to the developer to figure out. Any thought given to designing more…

My experience is that going from bash to something without a compiler has an overhead that rarely pays off. mypy or TS can probably do the job well, but at that point Rust gives you a better deployment story.
Post reply on HN