Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

51–60 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#51

Post by moreati »

TheDong wrote: post #47
moreati wrote: post #43

Instead of set -euf -o pipefail please consider set -o errexit set -o nounset set -o noglob set -o pipefail which is easier to lookup/search for if the reader is less familiar with shell scripting, and makes for cleaner diffs when a flag is removed/added. Caveat: some of the longer forms might be Bashisms (e.g. not present in ksh, dash, etc.)

"which is easier to lookup/search for" From my experiment: Googling "set euf" (without quotes) -- 838,000 results, first 6 results are all helpful and explain it in detail. Googling "set o nounset" (without quotes) -- 38,700 results, entire first page seems helpful. I dunno, looks like they're identically easy to learn the meaning of.

"easier to understand/remember the meaning of" would have been better phrasing. I can remember/infer what `set -o noglob` does, but not `set -f`.

Re: Writing Safe Shell Scripts (2019)

#52

Post by chubot »

aidenn0 wrote: post #6
mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :)

Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left.

So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squigglies, they will say "fix this".

The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.

In other words, it was the typical "static analysis has annoying false positives" issue.

Most of the shell scripts I write deal with "trusted" filenames -- i.e. those checked into the repo. If you want to deal with untrusted filenames, then there are a lot of ugly techniques to do that.

So Oil is basically meant to make the default thing the right thing, and the common thing the short thing (Huffman coding, as Larry Wall calls it).

Re: Writing Safe Shell Scripts (2019)

#53

Post by chungy »

chousuke wrote: post #37
aidenn0 wrote: post #14

Earlier quoted context omitted.

With the "--" and assuming GNU rm, rm -f -- /path/foo* is safe. If you use a glob pattern that does not match itself, then you will want nullglob: rm foo.[cC] will remove a file named literally foo.[Cc] if neither foo.c nor foo.C exist, despite the fact that foo.[Cc] does not match the glob. nullglob is probably a good default for bashscripts, but I don't think it exists in posix.

Wouldn't the glob still have issues with files that contain spaces? My defensiveness generally takes a sharp upturn when I write scripts that will be run as root or ones that delete data, but I'll admit not trusting mktemp's exit code is probably taking it a bit too far.

glob expansions don't have issues with spaces. the shell just patches every full match as its own entry in the argument array.

Re: Writing Safe Shell Scripts (2019)

#54

Post by Pxtl »

pfranz wrote: post #26
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've primarily used Python 2 for 10+ years and I often find cases where shell scripts are preferable. The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time. A recent example was I needed to tar+split large files. `tar cf - -C / $filename | spli…

Yeah, most of that comes from the verbose process needed to invoke a process, right? That's something I noticed when going back and forth between PowerShell and C# - that if C# had clean support for invoking a process and collecting the results as an IEnumerable like PowerShell does, PS wouldn't really need to exist, since 90% of the time you're dropping into C#/.net objects to get anything done anyways.

Re: Writing Safe Shell Scripts (2019)

#56

Post by stebann »

gugagore wrote: post #50

I personally think "don't" is right. For those who are interested in trying to use Python instead, I found this to be a helpful resource: https://github.com/ninjaaron/replacing-bash-scripting-with-p...

I disagree, Python scripting will tie you to Python development uncertainties, while shell works with most operational environments.

Re: Writing Safe Shell Scripts (2019)

#58

Post by Pxtl »

heavyset_go wrote: post #30
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

PowerShell, mostly because PowerShell was designed as a mashup of Bash and C#.... And it's kind of a trainwreck in a lot of ways.

It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.

Re: Writing Safe Shell Scripts (2019)

#59

Post by jonahx »

chubot wrote: post #52
aidenn0 wrote: post #6

Earlier quoted context omitted.

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

> The double quotes are of course technically correct, but when you're writing a shell script in a subdir 8 levels deep in the repo that only operates on 3 files in that subdir, it's overkill and makes everything ugly.

Once it becomes habit the cost is zero, though.

And the alternative you're implicitly suggesting is: "Follow shellcheck except you are allowed to make an expert judgement and ignore it when you know it won't be dangerous and have determined that the code is unlikely to change in the future so that this exception will be safe going forward too."

I think it's clear that this isn't a practical rule on a team with many people -- especially where they are not all experts in shell, which is 100% of teams of many people.

Making the experts "add those annoying double quotes they don't really need in this case" is the far lesser of the two sins.

Re: Writing Safe Shell Scripts (2019)

#60

Post by rapfaria »

mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Using shellshock.net can actually teach you a bunch about scripting as well
Post reply on HN