Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

251–260 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#251

Post by dimitar »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Line-oriented pipelines are great and have their place but I'm still sticking to a high-level general purpose programming language (lets abbreviate this as HGPPL) for scripts longer than 10 lines, because the following reasons:

* I like to the HGPPL data structures and convenient library for manipulating them (in my case this is Clojure which has a great core library). Bash has indexed and associative arrays.

* Libraries for common data formats are also used in a consistent way in the HGPPL. I don't have to remember a DSL for every data format - i.e. how to use jq when dealing with JSON. Similarly for YAML, XML, CSVs, I can also do templating for configuration files for nginx and so on. I've seen way too many naive attempts to piece together valid YAML from strings in bash to know its just not worth doing.

* I don't want to switch programming language from the main application and I find helps "break down silos" when everyone can read and contribute to some code. If a team is just sysadmins - sure, make bash the official language and stick to it.

* I can write scripts without repeating myself using namespaces and higher-order functions, which my choice of paradigm for abstractions, others write cleanly with classes. You can follow best practices, avoid the use of ENV vars, but that requires extra discipline and it is hard to enforce on other for the type of places where bash is used.

Re: Shell script best practices, from a decade of scripting things

#252

Post by Spivak »

throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

> Only use that for bashisms where there's no POSIX alternative

This seems like really bad advice because the number of people writing bash massively massively outnumbers the people writing sh. Regex matching, glob matching, proper parsing, &&/||, no need to quote.

I would say the opposite, enjoy all the bashisms like (( )) [[ ]], numeric for loops, extended globs, brace expansion, ranges, OH GOD YES VARIABLE REFERENCES, and only rewrite when you absolutely have to make it work on sh.

Re: Shell script best practices, from a decade of scripting things

#253

Post by ndsipa_pomu »

throwaway2037 wrote: post #218
erlkonig wrote: post #157

Earlier quoted context omitted.

echo has long been unreliable. Even the built-in echo in the shells were unreliable in SunOS, because the shell would look at the binaries in your PATH and try to figure out whether to emulate the BSD vs SysV (IIRC) version of echo and then change what echo would do. So much for writing a single script (with echo) that would work for all your users on the same host. This is why you'll see code like this: echo 'prompt…

echo is unreliable; I agree. Instead, I use "paranoid" printf with leading double dash: prinf -- "fmt str here..." "$carefully" "$quoted" "$args"

printf -- "fmt str here..." "${carefully}" "${quoted}" "${args}"

Re: Shell script best practices, from a decade of scripting things

#255

Post by lupire »

jd3 wrote: post #195
Beltalowda wrote: post #19

> Use bash. Using zsh or fish or any other, will make it hard for others to understand / collaborate. Among all shells, bash strikes a good balance between portability and DX. I think fish is quite a bit different in terms of syntax and semantics (I'm not very familiar with it), but zsh is essentially the same as bash except without most of the needless footguns and awkwardness. zsh also has many more advanced featur…

I just use whatever the default is, and since that's zsh on macOS now, I just ported my ~/.profile to be functionally interchangeable between zsh and bash Same PS1, aliases, functions, etc. but with a couple of slight variations due to syntax differences

There is no default scripting language, though.

Re: Shell script best practices, from a decade of scripting things

#256

Post by Spivak »

kotlin2 wrote: post #250
Spivak wrote: post #246

Earlier quoted context omitted.

I mean they're not that bad. declare -A mydict( [lookma]=initalization ) mydict[foo]=bar echo "${mydict[foo]}" list=() list+=(foo bar baz) echo "${list[0]}"

Now do an associative array containing another associative array.

Sometimes a you just have to accept a language's limitations.

Try in Python to make a nested defaultdict you can access like the following.

    d = 
    d["a"]["b"]["c"]  # --> 42
Can't be done because it's impossible for user code to detect what the last __getitem__ call is and return the default.

Edit: Dang it, I mean arbitrary depth.

Re: Shell script best practices, from a decade of scripting things

#259

Post by tb_technical »

hnlmorg wrote: post #227
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> getting the shell quoting hell right Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest…

Off topic. What's your opinion on Python?

I also write shell scripts, but I'm just curious what you would think about a comparison.

Re: Shell script best practices, from a decade of scripting things

#260

Post by mrlemke »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

To add to this, it's designed to work in conjunction with small programs. You don't write everything using bash (or whatever shell) built-ins. It will feel like a crappier Perl. If there is some part of your script where you're struggling to use an existing tool (f.g. built-ins, system utils), write your own small program to handle that part of the stream and add it in to your pipe. Since shell is a REPL, you get instant feedback and you'll know if it's working properly.

It's also important to learn your system's environment too. This is your "standard library", and it's why POSIX compatibility is important. You will feel shell is limited if you don't learn how to use the system utilities with shell (or if your target system has common utilities missing).

As an example of flexibility, you can use shell and system utilities in combination with CGI and a basic web server to send and receive text messages on an Android phone with termux. Similar to a KDE Connect or Apple's iMessage.

Post reply on HN