Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

241–250 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#241

Post by Bost »

hiepph wrote: post #165
Beltalowda wrote: post #19

> Use bash. Using zsh or fish or any other, will make it hard for others to understand / collaborate. Among all shells, bash strikes a good balance between portability and DX. I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced featur…

I can't really stand Bash's arcane syntax, it drains my brain power (and time of consulting manual) every time I have to work with it. Switching to Fish has been a breath of fresh air for me. I think some people who want to use only Bash need to open their conservative mind. All of my personal shell scripts now are converted to Fish. If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Of c…

Bash as a language is a downright bad. Especially the mess around `,",'. Fish is better is in this regard, however the syntax of Fish's string-related functions is unbearable. (I have a growing suspicion, with string-related functions, syntactically valid expressions can be constructed, which don't compile!)

However, neither Bash nor Fish were created with Composability in mind, which is a show-stopper for me.

IMO don't use Bash if the script is longer than 20 lines and don't use Fish if it's longer than 50. Use Python. If you want to use a proper(!) language use any LISP-Dialect like Babashka, Guile Scheme, Racket, etc. If you need Types have a look at Haskell-Scripting.

EDIT: To clarify, use Fish for its bling-bling capabilities, don't use it for scripting and configuring your machine(s).

Re: Shell script best practices, from a decade of scripting things

#242

Post by jpitz »

jph wrote: post #15

I favor POSIX and dash over bash, because POSIX is more portable. If a shell script needs any kind of functionality beyond POSIX, then that's a good time to upgrade to a higher-structure programming language. Here's my related list of shell script tactics: http://github.com/sixarm/unix-shell-script-tactics

I see this argument a great deal here on hacker news. I think it's more important that the scripter understands their use case and develop accordingly. Posix portability is almost never a factor or the things that I develop, because we have a near monoculture in terms of operating system and version.

For me, the additional features that bash provides are much more important than a portability that I'll never need to use.

Re: Shell script best practices, from a decade of scripting things

#243

Post by j0hnyl »

ketanmaheshwari wrote: post #234
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

I really don't think so! If you have experience with any scripting, you can fully grok the fundamentals of awk in 1 hour. You might not memorize all the nuances, but you can establish the fundamentals to a degree that most things you would try to achieve would take just a few minutes of brushing up.

For those that haven't taken the time yet, I think this is a good place to start:

https://learnxinyminutes.com/docs/awk/

Of course, some people do very advanced things in awk and I absolutely agree that 1 hour of study isn't going to make you a ninja, but it's absolutely enough to learn the awk programming paradigm so that when the need arises you can quickly mobilize the solution you need.

For example: If you're quick to the draw, it can take less time to write an awk one liner to calculate the average of a column in a csv than it does to copy the csv into excel and highlight the column. It's a massive productivity booster.

Re: Shell script best practices, from a decade of scripting things

#244

Post by ducktective »

synergy20 wrote: post #225
ducktective wrote: post #190

Earlier quoted context omitted.

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

new to 'rq', it's not in active development, any other alternatives? it seems doing a lot other than convert structured data to json.

Not sure what it is doing more...I'm referring to this rq: https://github.com/dflemstr/rq#format-support-status

It converts to/from the listed formats.

There is also `jc` (written in Python) with the added benefit that it converts output of many common unix utilities to json. So you would not need to parse `ip` for example.

https://github.com/kellyjonbrazil/jc#parsers

Re: Shell script best practices, from a decade of scripting things

#245

Post by Spivak »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Also the fact that $() invokes a supparser which lets use double quotes in an already double quoted expression is something I miss when using Python-f strings.

Re: Shell script best practices, from a decade of scripting things

#246

Post by Spivak »

strunz wrote: post #230
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Eh, as soon as you have to deal with arrays and hash tables/dicts or something like JSON, bash becomes very painful and hard to read.

I mean they're not that bad.

    declare -A mydict( [lookma]=initalization )
    mydict[foo]=bar
    echo "${mydict[foo]}"

    list=()
    list+=(foo bar baz)
    echo "${list[0]}"

Re: Shell script best practices, from a decade of scripting things

#247

Post by Spivak »

bitofhope wrote: post #197
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

There are workloads where shell scripts are the so-called right tool for a job . All too often I see people writing scripts in "proper" languages and calling os.system() on every other line. Shell scripts are good for gluing programs together. It's fine to use them for that.

For me it's once you make switch to a "proper" language you realize how much lifting pipelines do when it comes to chaining external binaries together.

Re: Shell script best practices, from a decade of scripting things

#248

Post by Cerium »

throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

"Use [[ ]] for conditions"

Oh how I hate the double square bracket. It is the source of many head scratching bugs and time wasted. "The script works in my machine!" It doesn't work in production where we only have sh. It won't exit due to an error, the if statement will gobble the error. You only find the bug after enough bug reports hit that particular condition.

After a couple shots to the foot I avoid double square brackets at all cost.

Re: Shell script best practices, from a decade of scripting things

#249

Post by s28l »

throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

> Some scripts are better to either have explicit error handling code, or simply never fail.

Silently ignoring sub-commands that exit with a non-zero code is not the same thing as "never failing". Your script might return 0, but that doesn't mean it did what you expect it to.

> Also, don't use set -o nounset when set -u will do.

`set -o nounset` is a lot easier to understand for the next person to read the script. Yes, you can always open the manpage if you don't remember, but that is certainly less convenient than having the option explained for you.

What shell are you using that doesn't support `set -o nounset`? Even my router (using OpenWRT+ash) understands the long-form version.

> Only use that for bashisms where there's no POSIX alternative

I totally disagree. You expect people to know the difference between `[[ ... ]]` and `[ ... ]` well enough to know what the bash version is required? You expect the next person to edit the script will know that if they change the condition, then they might need to switch from `[` to `[[`?

How do you even expect people to test which of the two that they need? On most systems, `/bin/sh` is a link to `/bin/bash`, and the sh-compatibility mode of bash is hardly perfect. It's not necessarily going to catch a test that will fail in `ash` or `dash`.

I think the "YAGNI" applies to trying to support some hypothetical non-bash shell that more than 99% of scripts will never be run with. Just set your shebang to `#!/bin/bash` and be done with it.

I totally agree about `pipefail`, though. I got burned by this with condition like below: ``` if (foo | grep -Eq '...'); then ```

Since `-q` causes grep to exit after the first match, the first command exited with an error code since the `stdout` pipe was broken.

Re: Shell script best practices, from a decade of scripting things

#250

Post by kotlin2 »

Spivak wrote: post #246
strunz wrote: post #230

Earlier quoted context omitted.

Eh, as soon as you have to deal with arrays and hash tables/dicts or something like JSON, bash becomes very painful and hard to read.

I mean they're not that bad. declare -A mydict( [lookma]=initalization ) mydict[foo]=bar echo "${mydict[foo]}" list=() list+=(foo bar baz) echo "${list[0]}"

Now do an associative array containing another associative array.
Post reply on HN