Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

191–200 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#191
post #128

More opinions 1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for. 2. Never use…

2. Why should bash be in /usr/bin? Mine's in /usr/local/bin and I've seen vendored bash binaries in very weird places. Respect the user's PATH.

8. '[' and '[[' are both bash builtins.

9/15. If you're using shellcheck, you'll need to quote (almost) all vars anyway.

Re: Shell script best practices, from a decade of scripting things

#192

More than one decade of shell script: bash is not shell, and talking about bash without version is suspicious. I won't check with for version those tips applies, and continue writing POSIX shell as much as can. I might check which or those suggestions are POSIX, though.

Sure, but everyone has bash installed already and it's far more featureful than POSIX shell.

Any reason to avoid writing bash scripts, other than purism?

Re: Shell script best practices, from a decade of scripting things

#193

I agree with basically all of this. A few more: The order of commandline args shouldn't matter. Env vars are better at passing key/value inputs than commandline arguments are. Process-substitution can often be used to avoid intermediate files, e.g. `diff some-file temp; diff some-file temp` If you're making intermediate files, make a temp dir and `cd` into that - Delete temp dirs using an exit trap (more reliable tha…

> - It may be useful to copy `$PWD` into a variable before changing directory

$OLDPWD is set when you 'cd'. Also 'cd -' will take you back to the last directory.

https://pubs.opengroup.org/onlinepubs/9699919799/

Re: Shell script best practices, from a decade of scripting things

#194

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> "If you have to write more than 10 lines, then use a real language" I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.

There are three numbers in this industry: 0, 1 and infinity. Any other number - especially when stated as a rule, limitation, or law - is highly suspect.

Re: Shell script best practices, from a decade of scripting things

#195

> Use bash. Using zsh or fish or any other, will make it hard for others to understand / collaborate. Among all shells, bash strikes a good balance between portability and DX. I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced featur…

I just use whatever the default is, and since that's zsh on macOS now, I just ported my ~/.profile to be functionally interchangeable between zsh and bash

Same PS1, aliases, functions, etc. but with a couple of slight variations due to syntax differences

Re: Shell script best practices, from a decade of scripting things

#196

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

I also like Bash - it's a powerful language, especially when combined with a rich ecosystem of external commands that can make your life easier, e.g. GNU Parallel.

Handling binary data can also work in Bash, provided that you just use it as a glue for pipelines between other programs (e.g. feeding video data into ffmpeg).

One time, while working on some computer vision project, I had a need to hack up a video-capture-and-upload program for gathering training data during a certain time of day. It took me about 20 minutes and 50 lines of Bash to setup the whole thing, test it, and be sure it works.

Re: Shell script best practices, from a decade of scripting things

#197

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

There are workloads where shell scripts are the so-called right tool for a job. All too often I see people writing scripts in "proper" languages and calling os.system() on every other line. Shell scripts are good for gluing programs together. It's fine to use them for that.

Re: Shell script best practices, from a decade of scripting things

#198
post #165

> Use bash. Using zsh or fish or any other, will make it hard for others to understand / collaborate. Among all shells, bash strikes a good balance between portability and DX. I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced featur…

I can't really stand Bash's arcane syntax, it drains my brain power (and time of consulting manual) every time I have to work with it. Switching to Fish has been a breath of fresh air for me. I think some people who want to use only Bash need to open their conservative mind. All of my personal shell scripts now are converted to Fish. If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Of c…

Not going to install fish on all of my servers just so i can run your scripts, sorry. They already have bash pre-installed, though.

> If I want to run some POSIX-compatible script then I just use `bash scripts.sh`

Shouldn't you be using a shebang?

Re: Shell script best practices, from a decade of scripting things

#199

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility.

I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell.

The power of shell isn't in the scripting language itself, it's in how fluidly it lets you embed snippets of tr, sed, awk, jq, and whatever else you need.

And, critically, these languages callable from shell were not all there when shell was designed. The extension interface of spawning processes and communicating with arguments and pipes is just that powerful. That's where shell shines.

Re: Shell script best practices, from a decade of scripting things

#200
Naming your executable shell scripts with .sh has similar problems to Hungarian notation. If your ~/.local/bin shell script ends up useful in a lot of places, you may want to re-write it in something less crap (and I say that as an experienced bash abuser who knows it quite well and uses it a lot more than he should) than bash. When you do that, your python/nim/lua/whatever script now has .sh at the end. What was the point?

.sh is appropriate for a shell library module which you source from another shell script. It is not really appropriate for something which is more abstract (such as a "program" inside your PATH).

set -e / set -o errexit will only be helpful if you fundamentally understand exactly what it does, if you don't, you are bound to end up with broken code. Once you fundamentally understand set -e you will be better placed to decide whether it is appropriate to use it or more appropriate to simply do proper error handling. The oft repeated mantra of using set -e is really misleading a lot of people into thinking that bash has some sane mode of operation which will reduce their chance of making mistakes, people should never be mislead to think that bash will ever do anything sane.

set -u / set -o nounset breaks a lot of perfectly sensible bash idioms and is generally bad at what proponents of it claim it will help solve (using unset variables by accident or by misspelling). There are better linters which solve this problem much better without having to sacrifice some of what makes bash scripts easier to write/read.

set -o pipefail is not an improvement/detriment, it is simply changing the way that one of bash's features functions. pipefail should only be set around specific uses of pipelines when it is known that it will produce the intended result. For example, take this common idiom:

    if foo | grep -q bar; then ...
The above will NOT behave correctly (i.e. evaluate to a non-zero exit code) if grep -q closes its input as soon as it finds a match and foo handles the resulting SIGPIPE by exiting with a non-zero status code.

Guarding set -x / set -o xtrace seems unnecessary, -x is already automatically inherited. Just set it before running the program.

Good advice on using [[ but it is important to fundamentally understand the nuances of this, quoting rules change within the context of [[.

Accepting h and help seems incredibly unnecessary. If someone who has never used a unix-like operating system happens upon your script then they may find it useful. But I don't think catering to such a low common denominator makes sense. Your script should just handle invalid arguments by printing a usage statement with maybe a hint of how to get a full help message.

I'd say changing to your script's directory is almost never appropriate.

Shellcheck, while useful, is useful only if you understand bash well.

The lesson here is that if you think that you have spent enough time writing bash to suggest best practices, you've not spent enough time writing bash. Only when you realise that the best practice is to not use bash have you used bash long enough (or short enough).

If you want to write a script which you're going to rely on or distribute, learn bash inside out and then carefully consider if it's still the right option.

If you are unwilling or unable to learn bash inside out then please use something else.

Do not be fooled into thinking that some "best practices" you read online will save you from bash.

Post reply on HN