Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to write idempotent Bash scripts

arslan.io

Post reply on HN
Print view View original

151–160 of 195 posts

Re: How to write idempotent Bash scripts

#151

Post by jmts »

godelski wrote: post #133
united893 wrote: post #33

This is generally terrible advice. A better option is to 'set -e' and ensure that the bash script exits when there's a failure. Bash scripts can't be idempotent because they operate in an external environment that can't ever be. The better option is to just be extra safe. See: > http://redsymbol.net/articles/unofficial-bash-strict-mode/

I'm not sure why people see errors as a bad thing. Errors are good. They stop you from doing stupid things. But maybe this article is good for telling people why you should handle errors well or when you write code to make sure it fails gracefully. Because if you don't people are going to start telling others to use the force flag when removing (almost always a terrible idea). Learn to handle errors, don't learn to b…

My understanding of 'set -e' is to ensure that errors are handled appropriately, not to brute force around them. If an error occurs that is not handled, this is treated as a problem and the script exits to ensure that one failure doesn't cause additional unforeseen problems as a result of the script running to completion.

Re: How to write idempotent Bash scripts

#153

Post by 1337shadow »

jdgiese wrote: post #150
rb808 wrote: post #57

I think Python is getting so popular & ubiquitous now I can't see any reason to write bash scripts any more except for perhaps very simple ones that are just a few lines. Other people's bash scripts are regularly just too difficult and complex to maintain.

I agree, in general, but I do think it is worth pointing out that bash is very good at what it was made for---i.e., starting processes, redirecting their output, etc. Compare: { a; b && c; } ≫ log.txt With the same written in Python: from subprocess import check_call, CalledProcessError f = open("log.txt", "a+") try: check_call(["a"], stdout=f) except CalledProcessError as e: pass try: check_call(["b"], stdout=f) che…

That's why Xonsh exists !

Re: How to write idempotent Bash scripts

#154

Post by godelski »

jmts wrote: post #151
godelski wrote: post #133

Earlier quoted context omitted.

I'm not sure why people see errors as a bad thing. Errors are good. They stop you from doing stupid things. But maybe this article is good for telling people why you should handle errors well or when you write code to make sure it fails gracefully. Because if you don't people are going to start telling others to use the force flag when removing (almost always a terrible idea). Learn to handle errors, don't learn to b…

My understanding of 'set -e' is to ensure that errors are handled appropriately, not to brute force around them. If an error occurs that is not handled, this is treated as a problem and the script exits to ensure that one failure doesn't cause additional unforeseen problems as a result of the script running to completion.

The brute force method is the force flag (eg: `rm -f`). `set -e` is error handling. Force flag was constantly used in the post which is what I disagree with.

Re: How to write idempotent Bash scripts

#156

Post by dhimes »

haddr wrote: post #132
dredmorbius wrote: post #122

Earlier quoted context omitted.

diff -R You can use numerous tools to compare/manage branches; md5sum / shasum (md5 is usually useful, though not entirely safe), diff and kin, vimdiff, rsync, fdupes, jdupes, git, hg.

I went the „git” way once and I found it really cool! With all the goodies you get for free from git, like applying patches, checking differences, resetting to previous state etc.

Same, but with mercurial.

Re: How to write idempotent Bash scripts

#157

Post by cpitman »

mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

100% Agree. I originally got into Puppet after spending several months with a client that wanted me to write bash scripts to push out a major middleware deployment, switchover, and uninstall to 10,000+ stores. There was no standard for the hardware in the stores, and their configuration was all over the place because 100's of teams also used (non-idempotent) shell scripts to push changes.

I learned a lot about catching all the different bad things that can happen, adding prerequisite checks, rolling back changes automatically, etc. In the end, a tool like Puppet/Chef/Ansible would have taken a fraction of the time to develop scripts for.

Containers have definitely lessened the need for mutable configuration management, but they are still useful in the 90% of environments that are still working in a past decade.

Re: How to write idempotent Bash scripts

#158

Post by pjscott »

cjvirtucio wrote: post #19
Zenst wrote: post #4

Many people who write scripts do not factor the environment they are run in and miss including traps to handle events like being terminated mid-way thru and cleaning up to a sanitized state. After all, creating a file, you can't presume their is enough free space, but people do in scripts all the time. Then you have scripts that you want run once and if run again not have any more effect. FOr example a script to be r…

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

Apple's new-ish APFS filesystem includes something for this pattern: the renamex_np(2) system call with the RENAME_SWAP flag. This atomically swaps two files, and can be used to atomically swap your staging directory with the existing directory. I'm not sure if there's command line support yet, though.

Re: How to write idempotent Bash scripts

#159

Post by cyphar »

pjscott wrote: post #158
cjvirtucio wrote: post #19

Earlier quoted context omitted.

I have not been doing DevOps for very long, but a useful trick I've found is to create a staging directory, perform my work there, diff the result with the existing state, and copy if a difference exists. I've found it very easy to clean up to a sanitized state in this way; the existing state is safe, since all I have to do is just destroy my work.

Apple's new-ish APFS filesystem includes something for this pattern: the renamex_np(2) system call with the RENAME_SWAP flag. This atomically swaps two files, and can be used to atomically swap your staging directory with the existing directory. I'm not sure if there's command line support yet, though.

Linux has something similar with RENAME_EXCHANGE. Unfortunately MV(1) doesn't appear to have support for any of the renameat2(2) flags.

Re: How to write idempotent Bash scripts

#160

Post by ajxs »

pas wrote: post #22
mnutt wrote: post #11

It’s good to know which command line tools are idempotent, but I think once you start running automated scripts repeatedly you’re better off with something like Chef where you declaratively define the end state and let chef figure out how to converge upon that state. Taking the “create a directory” example, you would say “I want to end up in a state where this directory is created and has these permissions” and chef…

Chef isn't magic, it just uses a lot of cookbooks which are just running things to check what the state is. Usually those cookbooks will want to have things in a particular way, which might not be the way you want. Chef/Ansible/Puppet all have the problem of having many layers of overhead for the same thing. This makes them slow, hard to debug, hard to read and even write (in my opinion). Sure, bash is hell, but the…

Many of the comments in this thread seem to be sucked into a false dichotomy. There are better options than _either_ Rust or Bash for writing complicated shell scripts. Perl/Python are more suited to the task than either. The challenge of deciding what degree of complexity warrants the investment of more complicated language is a design decision left to the developer to figure out. Any thought given to designing more resilient software is a good thing, but I can't help thinking that expecting any real degree of safety and security from bash is expecting too much. I don't work in devOps mind you, but whenever I have to write scripts that do anything more dangerous than creating files I leave bash on the shelf.
Post reply on HN