Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

361–370 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#361

Post by gtowey »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

As someone who has used a lot of shell over my career, I do love it as a utility and a programming paradigm.

However the biggest issues I've had is that the code is really hard to test, error handling in shell isn't robust, and reusability with library type methods is not easy to organize or debug.

Those are deal breakers for me when it comes to building any kind of non trivial system.

Re: Shell script best practices, from a decade of scripting things

#362

Post by ndsipa_pomu »

palunon wrote: post #333
NateEag wrote: post #327

Earlier quoted context omitted.

Which works not just to preserve the previous statement from internal inconsistency, but also in regards to the incredibly useful Rule of Three ( https://en.m.wikipedia.org/wiki/Rule_of_three_(computer_prog... ).

> Which works not just to preserve the previous statement from internal inconsistency It doesn't. You now have 4 numbers.

0, 1, 3, 4 and infinity - there's four numbers in this industry.

Five There's five numbers in this industry 0, 1, 3, 4, 5 and infinity

Wait, I'll come in again

Re: Shell script best practices, from a decade of scripting things

#363

Post by henrydark »

foobarian wrote: post #276
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.

I often find that downloading lots of data from s3 using `xargs aws sync`, and then xargs on some crunching pipeline, is much faster than a 100 core spark cluster

Re: Shell script best practices, from a decade of scripting things

#364

Post by ndsipa_pomu »

pferde wrote: post #345
blobbers wrote: post #338

As someone who used to have to write a lot of shellscripts because I worked at a company that believed in files and not databases, if you want to get funky use: shellcheck It's like pylint for your shellscripts.

A thousand times this. Shellcheck is a godsend that will save you tons of headaches if you have to deal with longer shell scripts - whether you are writing new scripts or maintaining old ones.

Job interviews should require unix admins to write a ten line bash script that passes shellcheck on the first attempt.

Also, write a "find" command without checking the manpage or internet

Re: Shell script best practices, from a decade of scripting things

#365

Post by throwawaaarrgh »

s28l wrote: post #249
throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

> Some scripts are better to either have explicit error handling code, or simply never fail. Silently ignoring sub-commands that exit with a non-zero code is not the same thing as "never failing". Your script might return 0, but that doesn't mean it did what you expect it to. > Also, don't use set -o nounset when set -u will do. `set -o nounset` is a lot easier to understand for the next person to read the script. Ye…

> Silently ignoring sub-commands that exit with a non-zero code is not the same thing as "never failing".

Well I meant the former. Very useful for things like init scripts where you would prefer the script do as much as it can to get something online.

> What shell are you using that doesn't support `set -o nounset`?

You're right, this does appear to be in POSIX, so I guess it's fine. But it is unusual to see in my experience.

> You expect people to know the difference between `[[ ... ]]` and `[ ... ]` well enough to know what the bash version is required?

No, I want them to use POSIX semantics until they have to do something Bash-y. Simplicity when it doesn't cost anything extra is best practice.

> Just set your shebang to `#!/bin/bash` and be done with it.

Homebrew, Jenkins, Asdf, etc may provide their own version of Bash that is required rather than the system default, and some systems have no /bin/bash at all. So you should use #!/usr/bin/env bash for Bash scripts and #!/usr/bin/env sh for POSIX Shell scripts. This lets the user override the PATH with their required version of Bash for this script. (and the script itself can check for versions of Bash, and even re-exec itself)

Re: Shell script best practices, from a decade of scripting things

#366

Post by ndsipa_pomu »

tuvi13 wrote: post #226

I get my "best practices" from here: https://tldp.org/LDP/abs/html/index.html I think this site is amazing, and it must be older than at least two decades.

I used to refer to that all the time, but it doesn't have newer bashisms (shell != bash).

A better resource is https://mywiki.wooledge.org/BashGuide Also, a preliminary read of https://mywiki.wooledge.org/BashPitfalls is advised.

Using shellcheck as a bash/shell linter is the ultimate. When you get a new warning, you can look up the code and learn why it's complaining.

Re: Shell script best practices, from a decade of scripting things

#368

Post by ndsipa_pomu »

throwaway2037 wrote: post #229

I'm surprised that no one mentioned a pair of tiny functions to log each command before it is run. Nicer versions also print a timestamp. Of course, this setup assumes: set -e echo_command() { echo echo '$' "$@" } echo_and_run_command() { echo_cmd "$@" "$@" } Then something like: main() { # For simple commands that do not use | etc. echo_and_run_command cp --verbose ... # More complex commands echo_command grep ... '…

I feel like I'm spamming these comments with this, but check out https://bash3boilerplate.sh/ for a much better logging system along with a neat way of parsing options. You define the usage and help section like so to define your options:

  ### Usage and help - change this for each script
  ##############################################################################

  # shellcheck disable=SC2015
  [[ "${__usage+x}" ]] || read -r -d '' __usage
Then you get to refer to ${arg_t} for the --timestamps option etc.

Re: Shell script best practices, from a decade of scripting things

#370

Post by HyperSane »

ndsipa_pomu wrote: post #328
HyperSane wrote: post #296

Earlier quoted context omitted.

"You have much less options when you are processing PowerShell objects." There is simply no need for the kind of extensive text processing common in Linux because every command returns an object whose fields can be directly referenced. Combined with the ConvertTo-Json command this is incredibly powerful. Honestly it seems like you are attempting to do things in PowerShell the bash way instead of the PowerShell way. "…

I tried to write a PowerShell script to recursively scan and find files/folders older than a certain date, but kept hitting problems with the length of the path/filenames. As a complete PowerShell noob, I'm sure I was trying to do it the wrong way, but after a few attempts, I gave up and install cygwin instead.

You can enable long path support in Windows to have paths up to 32,767 characters long

https://learn.microsoft.com/en-us/windows/win32/fileio/maxim...

Post reply on HN