Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

201–210 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#201

Post by ducktective »

Beltalowda wrote: post #23
jph wrote: post #15

I favor POSIX and dash over bash, because POSIX is more portable. If a shell script needs any kind of functionality beyond POSIX, then that's a good time to upgrade to a higher-structure programming language. Here's my related list of shell script tactics: http://github.com/sixarm/unix-shell-script-tactics

I've rewritten a lot of shell scripts with awk. Obviously it's not a good fit for everything, but when it is a good fit I found it a very pleasant experience. In spite of using Unix systems for 20 years I only learned awk a few years ago and I really beat myself up for not learning it earlier.

Convince me to up my game in awk!

I only use it to select the n'th word in a csv-like line. Anything more than that, I need to search stackoverflow for the invocation.

Don't you find its syntax cumbersome?

Re: Shell script best practices, from a decade of scripting things

#202

Post by rascul »

ilikejam wrote: post #191
casey2 wrote: post #128

More opinions 1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for. 2. Never use…

2. Why should bash be in /usr/bin? Mine's in /usr/local/bin and I've seen vendored bash binaries in very weird places. Respect the user's PATH. 8. '[' and '[[' are both bash builtins. 9/15. If you're using shellcheck, you'll need to quote (almost) all vars anyway.

> 8. '[' and '[[' are both bash builtins.

'[' is a builtin, '[[' is a keyword. Can use bash's builtin 'type' to check.

Re: Shell script best practices, from a decade of scripting things

#203

Post by throwawaaarrgh »

"Use set -o errexit"

Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior.

"Prefer to use set -o nounset."

ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". There is no real downside.

"Use set -o pipefail."

Waste of time. You will spend so much time debugging your app from random pipe failures that actually didn't matter. Dont use this option; just check the output of the pipe for sane values.

"Use [[ ]] for conditions"

No!!! Only use that for bashisms where there's no POSIX alternative and try to avoid them wherever possible. YAGNI!

"Use cd "$(dirname "$0")""

Use either "$(dirname "${BASH_SOURCE[0]}")" or grab a POSIX readfile-f implementation.

"Use shellcheck."

This should have been Best Practice #1. You will learn more about scripting from shellcheck than 10 years worth of blog posts. Always use shellcheck. Always.

Also, don't use set -o nounset when set -u will do. Always avoid doing something "fancy" with a Bashism if there's a simpler POSIX way. The whole point of scripts is for them to be dead simple.

Re: Shell script best practices, from a decade of scripting things

#204

Post by snidane »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes.

What Python is to Java, Shell is to Python. It speeds you up several times. I started using inline 'python -c' more often than the python repl now as it stores the command in shell history and it is then one fzf search away.

While neither Shell or SQL are perfect, there have been many ideas to improve them and for sure people can't wait for something new like oil shell to get production ready, getting the shell quoting hell right, or somebody fixing up SQL, bringing old ideas from Datalog and QUEL into it, fixing the goddamn NULL joins, etc.

But honestly, nothing else even comes close to this 10x productivity increase over the next best alternative. No, Thank you, I will not rewrite my 10 lines of sh into python to explode it into 50 lines of shuffling clunky objects around. I'll instead go and reread that man page how to write an if expression in bash again.

Re: Shell script best practices, from a decade of scripting things

#205

Post by shifto »

Beltalowda wrote: post #66
pizza234 wrote: post #31

Earlier quoted context omitted.

I don't know fish, but I don't consider zsh a step in the right direction, as it tries to be just a cleaned up Bash, which is not enough. There is a general problem in the fact that a radical evolution of glue languages wouldn't be popular because devs rather use Python, and small evolutions wouldn't be popular (ie. zsh), because they end up being confusing (since they're still close to Bash) and not bringing signifi…

zsh is not a "cleaned-up bash"; it's more of a clone of ksh (closed source at the time), with some csh features added in, as well as their own inventions. bash and zsh appeared at roughly the same time, many features were added in zsh first and added to bash later (sometimes much later, and often never). This is kind of a good example of what I meant when people conflate "bash" with "shell". As for your larger point:…

Just so you know. You can abbreviate almost anything in powershell or make your won aliases. I love Powershell, hands down best investment in my personal career was to really learn and understand Powershell.

Re: Shell script best practices, from a decade of scripting things

#206

Post by j0hnyl »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Re: Shell script best practices, from a decade of scripting things

#207

Post by xeddit »

ndsipa_pomu wrote: post #121

I think BASH scripting is the opposite of riding a bike - you end up re-learning it almost every time you need to do it

What really put a stick in my spokes early on was not realising how whitespace acts differently to what I was used to. syntax error near unexpected token ? I was missing a space inside a [[ ]] - I started paying more close attention, this isn't javascript.

Re: Shell script best practices, from a decade of scripting things

#208

Post by chasil »

kqr wrote: post #199
ducktective wrote: post #190

Earlier quoted context omitted.

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

I think this comment points to an even deeper insight: shell is a crappy programming language but with amazing extensibility. I would argue that once you pull in jq, you're no longer writing in "shell", you're writing in jq, which is a separate and different language. But that's precisely the point! Look at how effortless it is to (literally) shell out to a slew of other languages from shell. The power of shell isn't…

The shell is an ambiguous language that cannot be directly implemented with an LR parser.

Perhaps some of the power emerges from that ambiguity, but it is quite difficult to implement.

This presentation sums up the woes of an implementor:

https://archive.fosdem.org/2018/schedule/event/code_parsing_...

Re: Shell script best practices, from a decade of scripting things

#209

Post by hdjjhhvvhga »

ndsipa_pomu wrote: post #134
jraph wrote: post #126

Earlier quoted context omitted.

> obviously you wouldn't just run something from the internet without checking it first This died a long time ago with the pervasive use of NPM and PIP and the likes. Most developers probably run a lot of random unchecked shit all the time with local user privileges today without a blink. Somehow people are ready for all this, but are still afraid to run a random shell script from the internet. I guess this fear is o…

It's also fairly common to use a docker container that someone else built without having a look at it

I don't know. I do that a lot on test servers when tinkering with new stuff but at work I'm very careful what I insert into my employer's infrastructure. If someone breaks in using a hole I should have taken care of, that's already bad. But if invite bad guys by installing a C&C for them, that's superbad.

Re: Shell script best practices, from a decade of scripting things

#210

Post by 0xFEE1DEAD »

> Use set -o errexit at the start of your script. [...]

A couple of days ago this link was posted to hn http://mywiki.wooledge.org/BashFAQ/105

It showed me once again how little bash I know even after all those years. I checked the examples to see if only set -e is dangerous or also set -o like the author suggested and sure enough it's just as bad es set -e. You just got to thoroughly check your bash scripts and do proper error handling.

Post reply on HN