Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

21–30 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#21

Post by aidenn0 »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Here's a comment on that from someone working to make a better bash:

https://www.oilshell.org/blog/2018/01/28.html#shouldnt-scrip...

Re: Writing Safe Shell Scripts (2019)

#22

Post by kick »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Shell scripts are readable by just about anyone, they're available on every UNIX system, not just the Red Hat/Debian-derivatives of the last twenty years, they're fast as long as you're not doing stupid things, they're easily maintainable, they don't handle dependencies terribly (unlike Python), and so forth.

There's a reason AT&T used to run ads that showed their secretaries, managers, and so on using and writing shell scripts and there's never been a Python ad claiming that just anyone could write it.

Re: Writing Safe Shell Scripts (2019)

#23

Post by dwheeler »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Different tools are better at different things.

Shell is required to be basically everywhere. It's part of POSIX. It's also standardized. Shell scripts written 30 years ago still work, and will probably work 30 years from now too. Shell is probably on your TV.

Not everyone has Python. The most common version in the field is Python2, but that's officially obsolete. Python3 is in many places, but not everywhere. The two Pythons are basically incompatible unless you're willing to put in an effort. You have to work to make Python scripts compatible with both versions, especially if you can only use built-ins, and the results are way clunkier than that 2-line shell script you're trying to replace.

Python is a terrible choice if your goal is a relatively short script that calls other programs. You can call out to other programs in Python, but it's much clunkier and more painful to maintain. And while it often doesn't matter (because neither are speedy), CPython 3 takes more than 50x more time to start than dash and 27x time to start than bash (see: https://github.com/bdrung/startup-time , which shows Python3 3.6.4 at 197.79 ms, Bash 4.4.12 at 7.31 ms, and dash 0.5.8 at 2.81 ms). That speed is helpful when you want to have a "simple script in a loop".

Python is much better than shell when you start needing more sophisticated data types, libraries, modularity, etc. But the typical use for a shell script doesn't need any of that. If you need that, shell is a terrible tool, because it's the wrong tool for the job, not because it's never useful.

If you're writing shell, use shellcheck. Once you do that, your error likelihood goes way down. Many of the reports of problems writing shell come from a time when shellcheck didn't exist.

Re: Writing Safe Shell Scripts (2019)

#24

Post by smartmic »

savolai wrote: post #16
smartmic wrote: post #3

Shell scripts have their well-deserved place in Unix systems. A general recommendation to use Python or other high-level scripting languages without a discussion about the reasons why and when to use shell may lead to wrong conclusions.

I'd love to read a good blog post on when to use shell scripts vs Python or other programming languages, which seem far more accessible to me.

I agree, there is not just one true answer to it. For me, I would prefer shell scripts in many bootstrapping processes targeted on system administration. Also as glue language for many GNU tools, shell (then probably bash) is indispensable. With pure shell I can reduce the number of abstraction layers and dependencies. Also POSIX cross-platform compatiblity is a big point, just think of ``./configure`` in software builds.

Re: Writing Safe Shell Scripts (2019)

#25

Post by TheDong »

alexis_fr wrote: post #18

At the beginning of each file: #!/bin/bash set -euf -o pipefail cd $(dirname $0) Then add "" everywhere ;) And they say, write Python instead, except I’m dubious because python programs can have a lot of dependencies which can be tricky to install.

Your recommendations are a little off in the following ways:

You should use /usr/bin/env bash for the shebang. Some distros, such as nixos, don't have /bin/bash.

You need more quoting in the dirname line. cd "$(dirname "$0")" is what you need. The outer quotes are in case the directory has a space or special character in it, the inner ones are in case the script does.

That being said, you may also want $BASH_SOURCE rather than $0, but the reasons behind that are complicated. You may also want to support cases where the script is a symlink depending on what you're doing.

Re: Writing Safe Shell Scripts (2019)

#26

Post by pfranz »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've primarily used Python 2 for 10+ years and I often find cases where shell scripts are preferable.

The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time.

A recent example was I needed to tar+split large files. `tar cf - -C / $filename | split --bytes ${size}MB --verbose - $tmpfile.` My pure-Python implementation used the Tarfile library and I wrote a custom file-like object to split it--at least 50 lines. Both methods have pros/cons, but I had both implementations handy depending on the context I needed it in.

Another recent example was something I wrote to merge multiple video file "parts" into a single video using ffmpeg. It's 4 lines of shell script and would at least be 3x as long to write in Python.

The rule of thumb I have is if it's longer than 20 lines reevaluate if a shell scripting language is appropriate and I haven't really seen a need to change that in ~15 years I've used it.

Re: Writing Safe Shell Scripts (2019)

#27

Post by walshemj »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

For anything that isn't super trivial and I mean super trivial < 10 lines - Perl is a better solution probably more standardised that Python.

Re: Writing Safe Shell Scripts (2019)

#28

Post by codys »

vbernat wrote: post #13

Since it's bash-specific, I think people should use zsh instead. No need to quote variables as by default, splitting is not done. And you get access to arrays and associative arrays.

Bash has arrays.

    x=(1 2)
    echo "${x[0]}"
And associative arrays

    declare -A x
    x[hi]=20
    x[bye]=30
    echo "${x[hi]}"
    for i in "${!x[@]}"; do
        echo "k: $i, v: ${x[$i]}"
    done

Re: Writing Safe Shell Scripts (2019)

#29

Post by kohtatsu »

sillysaurusx wrote: post #19
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Ok, go ahead and port all 200 of these scripts to Python please: https://github.com/shawwn/scrap PR's welcome. ;) Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash. Here's an example. I use `llbranch` all the time: https://github.com/shawwn/sc…

Cool repo, but the example is lame because a shell, or a git alias would do just fine.

A shell script that consists of a one-line exec isn't what people concerned about.

FWIW I'm a 99.9% POSIX masochist (probably closer to 4N)

Re: Writing Safe Shell Scripts (2019)

#30

Post by heavyset_go »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion.

Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to install and maintain other interpreters or runtimes on an embedded platform.

Even job management and parallelism are easier in Bash and GNU Parallel.

[1] https://www.tldp.org/LDP/abs/html/process-sub.html

Post reply on HN