Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

101–110 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#101

Post by viraptor »

chubot wrote: post #52
aidenn0 wrote: post #6

Earlier quoted context omitted.

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

> in the repo that only operates on 3 files in that subdir

It does today. Will it change tomorrow? Will someone rename one of the files to include a space? It sounds like people saying "I don't need to escape this value in a query, it comes from a static list and it's safe." Yes - it's safe right now...

Re: Writing Safe Shell Scripts (2019)

#102

Post by filesystemdude »

dwheeler wrote: post #23
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Different tools are better at different things. Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV. Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The tw…

This.

A Python script is not the answer when I'm trying to do something quick and dirty at the shell.

The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done.

It now works on every computer I own.

I love Python. I use it almost every day. But I'll be damned if I'm going to go rewrite every shell snippet I hack together into Python just because it exists.

Different problems, different solutions.

Re: Writing Safe Shell Scripts (2019)

#103

Post by im3w1l »

nh2 wrote: post #92
dwheeler wrote: post #23

Earlier quoted context omitted.

Different tools are better at different things. Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV. Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The tw…

These are fair points. POSIX shell being fixed forever though also means it will /never/ get "fixed". Shell's silent-failure-by-default, implicit-by-default, principle-of-most-surprise semantics will always be there, and they are /bad/. I have read lots of shell scripts, by many people, from the experienced UNIX greybeard to to the novice web developer, and they were almost all flawed. Pointing out the flaws, the dev…

Compared to a proper language, shellscripts work for 80% of inputs with 20% of the code.

Re: Writing Safe Shell Scripts (2019)

#104

Post by mturmon »

filesystemdude wrote: post #102
dwheeler wrote: post #23

Earlier quoted context omitted.

Different tools are better at different things. Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV. Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The tw…

This. A Python script is not the answer when I'm trying to do something quick and dirty at the shell. The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done…

Your observation about the seamless evolution of a basic shell script from a few lines typed interactively is very insightful.

This kind of automation adds a lot of value for low marginal effort and probably explains a lot of the short scripts I have laying around in directories.

Re: Writing Safe Shell Scripts (2019)

#105

Post by kortilla »

chubot wrote: post #82
silly_giraffe wrote: post #64

Earlier quoted context omitted.

I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.

Thank you! Although I have to say I do feel the tradeoff right now between having a strong design / globally consistent code vs. allowing local variation / a lot of people to contribute quickly. That is, allow people to get in, add their useful patch, their useful bit of knowledge, and get out, without caring at all about the rest of the program. I think Linux, git, and GNU are the prototypical examples of the latter…

The trick is separating out external facing (api/ux) consistency from code style/data-structure/etc consistency. It doesn’t really matter if you don’t have coherency across the whole project for the latter because that can easily be swapped out later if it’s even a problem.

Most single-person project maintainers have trouble relaxing the latter and end up staying a single-person project because of it.

Re: Writing Safe Shell Scripts (2019)

#106

Post by montyhallpy »

stebann wrote: post #56
gugagore wrote: post #50

I personally think "don't" is right. For those who are interested in trying to use Python instead, I found this to be a helpful resource: https://github.com/ninjaaron/replacing-bash-scripting-with-p...

I disagree, Python scripting will tie you to Python development uncertainties, while shell works with most operational environments.

That's a good point. Shell would not announce an EOL date for a specific version. Scripts that used to work 25 years ago, still work just the same. Python scripts that used to work 10 years ago, now need maintenance/porting, no matter how trivial, to make them work with Python 3.

Re: Writing Safe Shell Scripts (2019)

#107

Post by ahnick »

urda wrote: post #98
mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.

Sorry, but what employers block the use of GPL tools? Most employers that write proprietary software don't want GPL code in their code, but using a tool like shellcheck to perform static analysis on your code doesn't make that software that was analyzed now subject to the GPL, so why would they block it?

Re: Writing Safe Shell Scripts (2019)

#108

Post by fctorial »

chubot wrote: post #79
jonahx wrote: post #59

Earlier quoted context omitted.

> The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly. Once it becomes habit the cost is zero, though. And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it…

Yeah I don't disagree with that. If you're working in a big group with some shell scripts, and want to get stuff done, use ShellCheck. I just find it annoying, hence the new shell :) There's a lot more to Oil, but that's definitely one of the surface annoyances I want to fix. Although zsh actually does fix that, and for some reason I've almost never seen a zsh script.

Many oh-my-zsh plugins are just collection of functions, which I think could be considered scripts.

https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins

Re: Writing Safe Shell Scripts (2019)

#109

Post by notyourday »

stebann wrote: post #56
gugagore wrote: post #50

I personally think "don't" is right. For those who are interested in trying to use Python instead, I found this to be a helpful resource: https://github.com/ninjaaron/replacing-bash-scripting-with-p...

I disagree, Python scripting will tie you to Python development uncertainties, while shell works with most operational environments.

You probably have one operating environment. It is the environment that you wrote that script for. Generalization in 99 out of 100 is introduction of an unnecessary flow that won't ever be executed outside the test but it will eat time.

Re: Writing Safe Shell Scripts (2019)

#110

Post by arglebargle123 »

I've done a bit of porting bash scripts to posix recently and I've found the following boilerplate pretty useful:

    case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
That takes care of setting -eu for shells that don't support -o pipefail and pipefail for bash.
Post reply on HN