Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

91–100 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#91

Post by AlphaSite »

kick wrote: post #22
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Shell scripts are readable by just about anyone, they're available on every UNIX system, not just the Red Hat/Debian-derivatives of the last twenty years, they're fast as long as you're not doing stupid things, they're easily maintainable, they don't handle dependencies terribly (unlike Python), and so forth. There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing sh…

Having seen many professionals writing shell scripts. No they can’t.

Re: Writing Safe Shell Scripts (2019)

#92

Post by nh2 »

dwheeler wrote: post #23
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Different tools are better at different things. Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV. Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The tw…

These are fair points.

POSIX shell being fixed forever though also means it will /never/ get "fixed".

Shell's silent-failure-by-default, implicit-by-default, principle-of-most-surprise semantics will always be there, and they are /bad/.

I have read lots of shell scripts, by many people, from the experienced UNIX greybeard to to the novice web developer, and they were almost all flawed. Pointing out the flaws, the devlopers of each level were surprised by them. It is close to impossible to write correct shell script, no matter the level of expertise, because the language offers neither semantics nor tooling that foster correctness.

Forgetting `set -e`, forgetting that it gets turned back off in subshells, quoting mistakes, pipefail semantics, you have to have made many mistakes to even /discover/ that these pitfalls exist (or use shellcheck, which will find some, but certainly not all of them).

Shell is a language that looks simple on the surface, but has 1000s of exceptions you have to master to write even minimal non-bugged scripts.

The lack of any form of reasonable data structures and strings being the prevalent data types also do not help. For example, NixOS's linker wrapper script (written in bash) was accidentally quadratic because of quadratic append-to-strings (https://github.com/NixOS/nixpkgs/issues/27609); and that was despite NixOS having many competent bash programmers. When scripts grow, these things happen.(Note that shellcheck does /not/ point out quadratic string appends; it can only find obvious mistakes and this is hard to detect in general). Fixing it was very difficult, and was not understood by many. If you have a language like Python that offers reasonable set/dict data structures that people understand and use intuitively, this is trivially avoided.

Finally, POSIX does not help much when the main thing shell does is calling other programs, and there are no guarantees on what the APIs for those programs are. For example, I have encountered many shell scripts that themselves are technically "portable", but they use arguments to e.g. `rm` that do not work on e.g. busybox, thus crashing the boot processe and resulting in hours of time lost for hundreds of people.

By not using shell, we can support that the next SOMETHING-standard will NOT mandate shell, but something better.

Re: Writing Safe Shell Scripts (2019)

#93

Post by sirtoffski »

mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Couldn’t agree more! I also really enjoy shfmt, integrating both ShellCheck and shfmt had really helped me to save time and write safer, readable scripts.

https://github.com/mvdan/sh

Re: Writing Safe Shell Scripts (2019)

#94

Post by liveoneggs »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

until they don't and python 3 is there instead?

Re: Writing Safe Shell Scripts (2019)

#95

Post by shakna »

arendtio wrote: post #75

Some time ago I started to doubt if using 'set -e' is a good idea. I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work. For illustration, take a look at the following example: #!/bin/bash foo() { set -e fals…

If I have something I know can fail, and I want it to be able to, then calling set +e before it, and recalling set -e afterwards is the way I handle it.

But, in this case, because you're running it in a subcommand, you need to add pipefail.

    set -euo pipefail
This way of making sure nothing misbehaves also locks out unset vars, which can be surprising, but also helpful.

Re: Writing Safe Shell Scripts (2019)

#96

Post by zelly »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

The shell is the true user interface to the OS. I like to think of writing shell scripts as having some mouse clicking automation tool on a GUI shell. It should be viewed as just as kludgy of a solution. But there really isn't anything more convenient most of the time. Shell scripts are to be called by humans to automate behavior they otherwise would have had to do manually.

An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).

Re: Writing Safe Shell Scripts (2019)

#97

Post by chubot »

hkiely wrote: post #70
chubot wrote: post #52

Earlier quoted context omitted.

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

Which is best for simply catching typing errors? Shell check? I never saw the instillation instructions for oil.

The idea is that with Oil there will be better solutions to the problems that ShellCheck points out.

As mentioned, the default should be right in Oil. The defaults are wrong in shell, and ShellCheck is meant to alert you of that.

Oil should have a style linter / formatter, but if it's designed correctly it shouldn't need the analogue of ShellCheck, which is more about correctness.

Re: Writing Safe Shell Scripts (2019)

#98

Post by urda »

mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.

Re: Writing Safe Shell Scripts (2019)

#99

Post by filesystemdude »

urda wrote: post #98
mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.

> For many professionals your employer will often block / avoid GPL code, tools, and libraries.

Indeed, which is why Linux is never found in a professional setting. /s

Re: Writing Safe Shell Scripts (2019)

#100

Post by urda »

filesystemdude wrote: post #99
urda wrote: post #98

Earlier quoted context omitted.

Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0 license. For many professionals your employer will often block / avoid GPL code, tools, and libraries.

> For many professionals your employer will often block / avoid GPL code, tools, and libraries. Indeed, which is why Linux is never found in a professional setting. /s

1) The sarcasm isn't appreciated here on HN.

2) Not everything in the FOSS world is GPL'd.

3) There are plenty of users who have this restriction in their work and it is a real restriction to be aware of when discussing tooling.

4) That's just how the code is licensed and is related to the topic at hand.

Post reply on HN