How to do things safely in Bash (2018) • Hacker News Board

Re: How to do things safely in Bash (2018)

#31

Post by marcosdumay » Sat, Apr 03, 2021, 8:06 PM UTC

yeowMeng wrote: post #7

pwdisswordfish8 wrote: post #2

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for e…

> The [python] scripts are much faster... Pipelines are performant; many algorithms can be expressed as pipelines. Bash has many built-in string manipulation mechanisms which do not require a sub shell. Why do people who subscribe to the “avoid shell scripts at all costs” ideology feel so passionately about prescribing to others? I like python, but if I am in a rush it’s shell.

Pipelines are a kernel feature that can be created on Python too. You'll just need to abstract it in a function.

Re: How to do things safely in Bash (2018)

#32

Post by dragonwriter » Sat, Apr 03, 2021, 8:25 PM UTC

stormbrew wrote: post #14

pwdisswordfish8 wrote: post #2

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for e…

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more…

> I just wonder, really, what "a shell script" even is to you if not something where you need to "shell out to external processes"?

A replacement for a shell script may not itself be a shell script.

A fairly typical python-as-shell-replacement I have that (among other things) creates/manipulates test resources in AWS corresponding to git feature branches, shells out less than it would if it was a typical shell language (using boto3 rather than shelling to the AWS CLI, etc.) but still shells out to git. But it would be replacing a shell script even if I had a git-repository-interaction library that eliminated the need to shell out.

Re: How to do things safely in Bash (2018)

#33

Post by belter » Sat, Apr 03, 2021, 8:29 PM UTC

Every time I think of writing some Bash I spend some time here: https://mywiki.wooledge.org/BashPitfalls and then ...I use something else.

Re: How to do things safely in Bash (2018)

#34

Post by drran » Sat, Apr 03, 2021, 8:43 PM UTC

xonix wrote: post #24

I personally have found that AWK is a surprisingly good alternative to shells for scripts bigger than average. Why? 1. Portability (AWK is a part of POSIX) 2. Very clean syntax 3. Powerful associative arrays 4. Super-powerful string manipulations facilities 5. Easy shell interoperability As an example here is a simplistic build tool [1] I’ve developed in AWK. As you can check [2] it runs unchanged under Linux/macOS/W…

AWK is good, but Perl is better. Perl is good, but Python is cleaner. Python is good, but Go is better. Go is good, but Rust is faster. Rust is good, but shell is simpler. Shell is good, but AWK is better...

Re: How to do things safely in Bash (2018)

#35

Post by gdavisson » Sat, Apr 03, 2021, 8:49 PM UTC

I agree with almost everything in this guide, but it has a couple of recommendations that create new problems (while solving others):

The nullglob option ('shopt -s nullglob') makes things like 'for f in .txt' work right when there are no matching files, but make other commands like 'grep somepattern .txt' change their behavior in ways that can cause serious trouble. grep (and many other commands), given no files as input, will read from standard input (up to the EOF); if it's running interactively and input hasn't been redirected, this causes the script to hang for no discernible reason. If input has been redirected, it steals input that was presumably meant to be read by some other command. In my opinion, the problems this causes are more serious than what it solves.

The errexit option ('set -e' or 'shopt -s errexit') can both fail to exit when you expect/want it to (several such situations are described in the guide) and also exit when you don't expect/want it to. The guide mentions suppressing unexpected exits with '|| true', but it's not always obvious when this is needed, and it's not even consistent between versions of bash. There's a good parable about this (and some examples) at http://mywiki.wooledge.org/BashFAQ/105

I really don't like trying to predict and work around unpredictable features like this; I'd much rather deal with explicit error handling, like 'commandThatMightFail || { echo "Aaaargh" >&2; exit 1; }'

Re: How to do things safely in Bash (2018)

#36

Post by dang » Sat, Apr 03, 2021, 8:53 PM UTC

Discussed at the time:

Safe ways to do things in bash - https://news.ycombinator.com/item?id=17057596 - May 2018 (240 comments)

Re: How to do things safely in Bash (2018)

#37

Post by drran » Sat, Apr 03, 2021, 9:03 PM UTC

effie wrote: post #29

macintux wrote: post #16

Earlier quoted context omitted.

How many strongly typed languages are: + Installed everywhere + Even within 20% as concise as bash

Indeed. Python does not win over bash in either category, and it is slower. I wish there was a super-fast shell-like scripting language everywhere that could serve as a general programming language. Like Perl, but simpler rules and readable syntax. But there is nothing. Who would have thought, I may have to learn Perl eventually.

IMHO, we need more than one language in the shell: one to orchestrate launching of applications, plumbing, and processing of their inputs and outputs, another one for computations with integers, floats, complex values, arrays, matrices, another one for text processing and parsing, another one for argument parsing and default values, another one for error handling and recovery, another one for pattern matching and state machines, another one for object-oriented programming, another one for network programming, another one for record-oriented programming, another one for security and debugging, another one for container management, another one for system manipulation, upgrading, recovering, another one for documentation, and so on, with advanced way to share variables and data between these languages (environment variables on steroids).

It's not possible to implement such monstrosity as one simple, portable, small, fast, and secure binary.

Re: How to do things safely in Bash (2018)

#38

Post by renewiltord » Sat, Apr 03, 2021, 9:04 PM UTC

Embarrassingly, instead of fixing the code, I fix the data for bash. I just make sure my input is well-formed haha. I don't think I have any files or folders with spaces in them on my computer.

It may not be the way but it is a way.

Re: How to do things safely in Bash (2018)

#39

Post by lpapez » Sat, Apr 03, 2021, 9:14 PM UTC

renewiltord wrote: post #38

Embarrassingly, instead of fixing the code, I fix the data for bash. I just make sure my input is well-formed haha. I don't think I have any files or folders with spaces in them on my computer. It may not be the way but it is a way.

I tend to do this as well.

This is the way.

Re: How to do things safely in Bash (2018)

#40

Post by kubanczyk » Sat, Apr 03, 2021, 9:33 PM UTC

blabitty wrote: post #18

stormbrew wrote: post #14

Earlier quoted context omitted.

This, and I mean very specifically this with python always as the go-to replacement, comes up every single time anyone even utters the words "shell" or "bash" and every time all I can think is that the people saying it aren't writing the same kinds of shell scripts I ever see, use, or wind up writing. And every time I've ever seen something that's allegedly "a shell script replaced with a python script" it's way more…

I wonder too. Things like file and directory manipulations are way more complicated in native Python and I find the exceptions harder to troubleshoot than native commands. I'm not even going to get into pythons version and dependency hell, suffice it to say I have more confidence in a standard version of bash and standard utilities generally being present on a modern Linux system. What I've seen from pythonistas as r…

Here is a novel idea.

I am allowed to call 'mv' or 'cp' binaries from bash.

I am also allowed to call 'mv' or 'cp' binaries from python with subprocess!

(Edit: bad example binaries, 'df' or 'tar' would be probably better.)

I must choose not to install bash libraries for a bash script beyond builtins.

I can choose not to install python libraries beyond builtins.

It becomes a really simple tradeoff.

Nobody forces you to step out of python's stdlib. No extra dependencies. And you can still reap benefits of a proper programming language.