Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

481–490 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#481
> Check if the first arg is `-h` or `--help` or `help` or just `h` or even `-help`, and in all these cases, print help text and exit.

`-h` and `--help` are fine. But `help` and `h` should only display the help if the script has subcommands (like `git`, which has `git commit` as a subcommand). Scripts that don't have subcommands should treat `h` and `help` as regular arguments -- imagine if `cp h h.bak` displayed a help message instead of copying the file named "h"!

I wouldn't encourage `-help` for displaying the help because it conflicts with the syntax for a group of single-letter options (though if `-h` displays the help, there is no legitimate reason for grouping `-h` with other options).

And ideally scripts that support both option and non-option arguments should allow `--` to separate them (e.g. `rm -- --help` removes the file called "--help"). But parsing options is complicated and probably out of scope for this article.

> If appropriate, change to the script’s directory close to the start of the script. And it’s usually always appropriate.

This is very problematic if the script accepts paths as arguments, because the user would (rightly) expect paths to be interpreted relative to the original working directory rather than the script's location. A more robust approach is to compute the script's location and store it in a variable, then explicitly prepend this variable when you want paths to be relative to the script's location.

Re: Shell script best practices, from a decade of scripting things

#483

Earlier quoted context omitted.

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

Can you give an example of how you'd use rq in this pipeline? I'm not finding any good examples

curl -s "give.me.some/yaml" | rq --input-yaml --output-json | jq '.my.selected[1].field'

Re: Shell script best practices, from a decade of scripting things

#484

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Aye.. I've been saying for years that shell scripting is how I meditate, and I'm only mostly joking

Shell quoting though, Aieeee...

I find I have to shift gears quite substantially moving from shell or powershell to anything else...

"I'll just pipe the output of this function into.. oh, right"

Re: Shell script best practices, from a decade of scripting things

#485
post #165

Earlier quoted context omitted.

I can't really stand Bash's arcane syntax, it drains my brain power (and time of consulting manual) every time I have to work with it. Switching to Fish has been a breath of fresh air for me. I think some people who want to use only Bash need to open their conservative mind. All of my personal shell scripts now are converted to Fish. If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Of c…

Not going to install fish on all of my servers just so i can run your scripts, sorry. They already have bash pre-installed, though. > If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Shouldn't you be using a shebang?

>Shouldn't you be using a shebang?

Why would they use a shebang? `bash script.sh` works perfectly fine. Ever used a terminal?

Re: Shell script best practices, from a decade of scripting things

#486
post #298

Earlier quoted context omitted.

POSIX shell is the standard, not bash.

That ship has sailed, because busybox ash and dash continually implement some of bash features and semantics, which come from ksh. And OSH implements almost all of bash That is, The posix shell spec is missing a lot of reality. It’s not very actively maintained, unfortunately The canonical example is not having local vars, which basically every shell supports

I agree that POSIX is needing of a few small additions;

* local keyword * 2 or 3 variable expansion tricks (string replacement etc) * pipefail

But with those, I think the spec isn't too bad. I've been writing (imo) high-quality (and shellcheck compliant) shell scripts for a decade, and _always_ try to be as pendantic about being posix compliant where humanly possible. Sometimes things are a _little_ harder (no sarcasm here), but it's really quite doable once you get the hang of it.

You scripts then end up being far more portable, and as stated below, MUCH easier to read. The trick is, to write readable code (which is hard enough for most people, regardless the language anyway :p)

e.g. (a very simple example), and an addition to Shrikant's post, avoid the terse double ampersant 'and' (&&) and double pipe 'or' (||) operators, in normal calls. do e.g. ```sh if ! external_command; then do something with failure fi ``` Rather then `external_command || do something with failure`

These can get up becoming really hard to read once they become longer, where reading a stupid "if" statement is easy to comprehend. Readability over saving number of lines.

One important hint left forgotten, always make sure the last line of your file is `exit 0` or similar. This is more a 'weak' security feature. Prevent people from appending files and executing stuff instead. Gives a known exit point.

Another addition would be to actually favor single quotes for pure strings, and use double quotes where you expect the shell to do something with your string (true for most cases, but there's plenty of strings that benefit from single quotes, and hint the reader what to expect). Also integers should never be quoted (as it would turn then into strings), which shows a 'bad' example in the trace statement, you are now comparing the strings '0' and '1' rather then the number. Best use something easier to read in that case.

One thing I will pick up from this post for sure, is the trace hint, I'm adding that to my scripts, but probably a little more tunable.

```sh set -eu if [ -n "${DEBUG_TRACE_SH:-}" ] && \ [ "${DEBUG_TRACE_SH:-}" != "${DEBUG_TRACE_SH#"$(basename "${0}")"}" ] || \ [ "${DEBUG_TRACE_SH:-}" = 'all' ]; then set -x fi

echo 'Hello World'

exit 0 ``` Though I'll probably just settle for one of those keywords, I like 'all' the best atm. This would run the trace only if this special keyword is given, or if the variable contains the name of the script. I initially had `basename "${0%%.sh}"` but that makes things like `test,test1,test2` impossible, though that only helps if the extension is used ;)

While not needing be part of this list, I personally also always use functions, unless the script really is less then a handful lines, like google's bash-guide, always have a main etc.

In the end though I do disagree writing bash and bashisms. There's plenty of posts about that though; kind of like the whole C++ 'which version to use' discussion, where bash tends to be inconsistent in itself.

Shameless plug: See some of my older stuff here https://gitlab.com/esbs/bootstrap/-/blob/master/docker_boots...

Re: Shell script best practices, from a decade of scripting things

#487

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design).

So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something bigger that 10 lines.

However, I have to admit, I don't follow that advice myself...

Re: Shell script best practices, from a decade of scripting things

#488

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design). So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something big…

> The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design).

I guess you're referring to http://mywiki.wooledge.org/BashFAQ/105. Got recently hit by these as well.

Re: Shell script best practices, from a decade of scripting things

#489

Earlier quoted context omitted.

The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design). So if you don't care about error cases everything is fine, but if you do, it gets ugly really fast. And that is the reason why other languages are probably be better suited if you want to write something big…

> The biggest Issue is that error handling is completely broken in POSIX shell scripting (including Bash). Even errexit doesn't work as any normal language would implement it (One could say it is broken by design). I guess you're referring to http://mywiki.wooledge.org/BashFAQ/105 . Got recently hit by these as well.

Yes and my personal favorite: Functions can behave differently depending on, if they are being called from a conditional expression vs. from a normal context. Errexit has no effect if the function is called from a conditional expression.

Re: Shell script best practices, from a decade of scripting things

#490

This guy shells!

Hey Peter! It's so humbling to see you check out my blog! Your articles on awk and sed were a huge inspiration to me around 2008-09, and I super-looked up to you. Never have I imagined you would check out my blog one day! Thank you for all your work dude! Stay awesome.

^5
Post reply on HN