Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

121–130 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#121

Post by peterwwillis »

savolai wrote: post #16
smartmic wrote: post #3

Shell scripts have their well-deserved place in Unix systems. A general recommendation to use Python or other high-level scripting languages without a discussion about the reasons why and when to use shell may lead to wrong conclusions.

I'd love to read a good blog post on when to use shell scripts vs Python or other programming languages, which seem far more accessible to me.

If all you need to do is chain together external programs, set environment variables, redirect filehandles, iterate over files, etc with the tiniest bit of logic, and don't want to have to set up an execution environment or download something to do it, and want ultimate Unixy portability, and you want virtually anyone to be able to read and maybe modify it, you want shell.

If you need to do a very specific task that involves interpreting/modifying data structures/formats, if you have non-trivial logic, if you need to use a module, if you need to access an operating system primitive which Unix shells don't really expose, if there's no Unix tool that does what you want, or you just need more control/certainty/reliability, you want Python or something else.

Re: Writing Safe Shell Scripts (2019)

#122

Post by unlinked_dll »

filesystemdude wrote: post #102
dwheeler wrote: post #23

Earlier quoted context omitted.

Different tools are better at different things. Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV. Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The tw…

This. A Python script is not the answer when I'm trying to do something quick and dirty at the shell. The beauty of shell scripting is that it evolves seamlessly from trying to solve simple problems at my command prompt. I pipe a couple of things together, and then I realize I could use a loop and a few conditionals, and suddenly it makes sense to store this in file in case I want to do this again. Boom, program done…

True that. But everyone should keep in mind that script bloat is real. If you're not careful you'll end up like me, and accidentally have to maintain what is essentially npm but written entirely in bash.

Re: Writing Safe Shell Scripts (2019)

#123

Post by tjoff »

I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist?

I've been thinking that we are kind of stuck with it. Much like javascript.

So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint.

A big part of shellscripts though is having the source available. So maybe even have the translated, original code, and the generated shell output in the same file.

The source code on top (out of view from a shell interpreter) and the generated code below. This would allow it to be readable by anyone and executable by anyone, but modifying it would require the transpiler (which by itself would probably be pretty lightweight as well - certainly compared to python).

Certainly not perfect, but preferable to rediscovering the nuances and gotchas of shell-scripts every other day. Or maybe such a solution would only prolong the suffering and we should start all over.

Re: Writing Safe Shell Scripts (2019)

#124

Post by sjellis »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

IMO, the power of shell is really the ability to leverage command-line tools (e.g. git, curl, jq) with very little code. These tools are very fast, well-tested, feature-rich, and often easy to install in a reproducable way.

IMO, The inflection point where you should stop using shell is very low, though. The Google style guide for shell recommends that you should rewrite scripts once they are more than 100 lines, and I think that it probably too generous.

Re: Writing Safe Shell Scripts (2019)

#125

Post by Bnshsysjab »

tjoff wrote: post #123

I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist? I've been thinking that we are kind of stuck with it. Much like javascript. So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint. A big part of shellscripts though is having the source availab…

What’s wrong with python?

Re: Writing Safe Shell Scripts (2019)

#126

Post by tjoff »

Bnshsysjab wrote: post #125
tjoff wrote: post #123

I'm not sure I'm comfortable with python for a typical shell script. But then again, what alternatives exist? I've been thinking that we are kind of stuck with it. Much like javascript. So, we could go the route of typescript and have a translation layer and outputting shell scripts. I actually think that could work quite well from a technical standpoint. A big part of shellscripts though is having the source availab…

What’s wrong with python?

Feels like the wrong tool for the job with the whole virtual environment that has to be brought up.

Performance for shell scripts isn't super critical in my eyes but the startup time of python feels too much for something you might want to run in an inner-loop from find or something. Not sure if python IO performance is suitable for shell scripts either.

Also isn't nearly as universal as shell-scripts nor something you necessarily have/want on small systems.

But I will admit that I'm channeling prejudices and don't feel I have enough of a complete picture to say anything definite about it. Others in this thread touch upon it though.

Re: Writing Safe Shell Scripts (2019)

#127

Post by heavyset_go »

Pxtl wrote: post #58
heavyset_go wrote: post #30

Earlier quoted context omitted.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

PowerShell, mostly because PowerShell was designed as a mashup of Bash and C#.... And it's kind of a trainwreck in a lot of ways. It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.

Some of the features of PowerShell seem really enticing, and I wish they would make their way over to Unix-like shells. Unfortunately, I don't know how much utility I'd get out of PowerShell on Linux, so I haven't tried it.

edit: your comment inspired me so I installed PowerShell via Snap and am going to give it a go on Linux.

Re: Writing Safe Shell Scripts (2019)

#128

Post by heavenlyblue »

IshKebab wrote: post #65
heavyset_go wrote: post #30

Earlier quoted context omitted.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

I don't know about process substitution, but Dart makes connecting processes together pretty easy and you can do way more fancy things than Bash let's you do, in a clean fast language with basically no gotchas. Check out dshell: https://pub.dev/packages/dshell

And now we have 3 languages...

Re: Writing Safe Shell Scripts (2019)

#129

Post by arendtio »

shakna wrote: post #95
arendtio wrote: post #75

Some time ago I started to doubt if using 'set -e' is a good idea. I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work. For illustration, take a look at the following example: #!/bin/bash foo() { set -e fals…

If I have something I know can fail, and I want it to be able to, then calling set +e before it, and recalling set -e afterwards is the way I handle it. But, in this case, because you're running it in a subcommand, you need to add pipefail. set -euo pipefail This way of making sure nothing misbehaves also locks out unset vars, which can be surprising, but also helpful.

As far as I can tell, adding pipefail or nounset doesn't change the behavior in this case.

In my opinion, the problem is that calling a function from a test expression, evaluates it in a different manner than calling it from elsewhere. That is so absurd that I wonder how it wasn't changed over the years.

Re: Writing Safe Shell Scripts (2019)

#130

Post by arendtio »

arpa wrote: post #114
arendtio wrote: post #75

Some time ago I started to doubt if using 'set -e' is a good idea. I mean, if it would work as you expect it to, it certainly is a good idea to exit a script as soon as something fails. But sadly not all implementations behave similarly [1] and if you call a function from within a condition, 'set -e' gets deactivated/doesn't work. For illustration, take a look at the following example: #!/bin/bash foo() { set -e fals…

Obsessively checking for failures like people do in go is the "correct" way. Using && to chain groups of your commands that depend on each other also helps.

Well, I am not sure what the correct way of handling errors is. I like the concept of Exceptions, but not the results when people use them. Somehow they encourage devs to care less about error checking.

And while I appreciate the thoroughness of handling errors the Go way, I find it tiring and verbose. In many cases, you just want to exit with a specific exit code and adding 4 lines just to specify that exit code doesn't feel right.

The problem I have with errexit is that you can't rely on it. You can write your function with it (even enabling it explicitly), but later someone comes along, calls it in the wrong context and boom, your function runs without errexit.

Post reply on HN