Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

111–120 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#111

Post by 3xblah »

chubot wrote: post #52
aidenn0 wrote: post #6

Earlier quoted context omitted.

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

"The double quotes are of course technically correct, but when ... it's overkill and makes everyrthing ugly."

The only way you know this is being a person who routinely writes shell scripts.

There is no substitute for knowing what you're doing.

Try running shellcheck on configure scripts.

Or try this

    curl -4O https://ftp.netbsd.org/pub/NetBSD/NetBSD-release-8/src/build.sh
    shellcheck build.sh
Or try shellcheck on the build.sh from buildroot.org.

These scripts are "safe" enough for hundreds or thousands of competent users to be running them every day.

Re: Writing Safe Shell Scripts (2019)

#112

Post by arglebargle123 »

TheDong wrote: post #25
alexis_fr wrote: post #18

At the beginning of each file: #!/bin/bash set -euf -o pipefail cd $(dirname $0) Then add "" everywhere ;) And they say, write Python instead, except I’m dubious because python programs can have a lot of dependencies which can be tricky to install.

Your recommendations are a little off in the following ways: You should use /usr/bin/env bash for the shebang. Some distros, such as nixos, don't have /bin/bash. You need more quoting in the dirname line. cd "$(dirname "$0")" is what you need. The outer quotes are in case the directory has a space or special character in it, the inner ones are in case the script does. That being said, you may also want $BASH_SOURCE r…

Using 'env bash' to find the local path is definitely the way to go. It also supports that occasional case where the user has installed modern bash somewhere in their local path (under their home directory) and finds that rather than the system copy.

    #!/usr/bin/env bash
If you really want a predictable environment I usually go with something like this after the shebang, it supports bash and not-bash shells:

    case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
    PATH=/usr/sbin:/usr/bin:/sbin:/bin
    \unalias -a
I don't usually cd anywhere unless I need to.

Re: Writing Safe Shell Scripts (2019)

#113

Post by yyx »

3xblah wrote: post #111
chubot wrote: post #52

Earlier quoted context omitted.

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

"The double quotes are of course technically correct, but when ... it's overkill and makes everyrthing ugly." The only way you know this is being a person who routinely writes shell scripts. There is no substitute for knowing what you're doing. Try running shellcheck on configure scripts. Or try this curl -4O https://ftp.netbsd.org/pub/NetBSD/NetBSD-release-8/src/build.sh shellcheck build.sh Or try shellcheck on the…

>all I get are false positives

double quotes, mixing string and array, unused variables. Why is this normal?

Re: Writing Safe Shell Scripts (2019)

#114

Post by arpa »

arendtio wrote: post #75

Some time ago I started to doubt if using 'set -e' is a good idea. I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work. For illustration, take a look at the following example: #!/bin/bash foo() { set -e fals…

Obsessively checking for failures like people do in go is the "correct" way. Using && to chain groups of your commands that depend on each other also helps.

Re: Writing Safe Shell Scripts (2019)

#115

Post by arpa »

clarry wrote: post #11
emj wrote: post #10

Earlier quoted context omitted.

Please tell me it's enough to check the exit code! That's all I ever do and I have hundreds of mktemp scripts out there. In the end I think shelling is mostly about controlling your inputs.

Not just inputs, unfortunately. TOCTOUs (and races in general), for instance, are very common problem in shell scripts.

Races and TOCTOUs are generally common problems in most concurrent systems that are not considering atomicity by design (e.g. rdbms). It's just that shell can very easily become concurrent...

Re: Writing Safe Shell Scripts (2019)

#118

Post by sjy »

chrisfinazzo wrote: post #71
Pxtl wrote: post #58

Earlier quoted context omitted.

PowerShell, mostly because PowerShell was designed as a mashup of Bash and C#.... And it's kind of a trainwreck in a lot of ways. It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.

As I understand it, PowerShell came about because porting the standard Linux/Unix utilities to Windows didn't work out. PowerShell meets both requirements as it's tuned for the Windows environment, but can still play nicely with the rest of the world. Of course, now that WSL is a thing, it's a real question why someone would want to use PS unless they had no other options. https://www.heavybit.com/library/podcasts/to…

PowerShell has real data structures (arrays and hash tables), built-in functional programming tools (Select-Object, ForEach-Object, Where-Object), GUI cmdlets (Out-GridView) and direct access to .NET libraries. It’s a viable tool for many complex tasks where on Unix, you’d need to reach for Python rather than a shell.

Re: Writing Safe Shell Scripts (2019)

#119

Post by sjy »

kick wrote: post #22
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Shell scripts are readable by just about anyone, they're available on every UNIX system, not just the Red Hat/Debian-derivatives of the last twenty years, they're fast as long as you're not doing stupid things, they're easily maintainable, they don't handle dependencies terribly (unlike Python), and so forth. There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing sh…

> Shell scripts are readable by just about anyone

I reckon that less than 0.1% of the population can read shell scripts.

Re: Writing Safe Shell Scripts (2019)

#120

Post by [deleted] »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

[deleted]
Post reply on HN