Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

31–40 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#31

Post by rasguanabana »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

It’s a real pain to handle subprocesses in Python. If you need to automate certain command based workflows, Bash scripts are much easier, both to write and read, until they reach certain size. At my previous job it was a daily task and scripts involved tricky stuff related with Subversion, Git and builds (complex CI/CD, generally).

The problem is that very few people know shell scripting well, but once you get to know it, it’s not that bad, in quite specific cases.

Always use Shellcheck, though.

Re: Writing Safe Shell Scripts (2019)

#32

Post by kohtatsu »

kohtatsu wrote: post #29
sillysaurusx wrote: post #19

Earlier quoted context omitted.

Ok, go ahead and port all 200 of these scripts to Python please: https://github.com/shawwn/scrap PR's welcome. ;) Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash. Here's an example. I use `llbranch` all the time: https://github.com/shawwn/sc…

Cool repo, but the example is lame because a shell, or a git alias would do just fine. A shell script that consists of a one-line exec isn't what people concerned about. FWIW I'm a 99.9% POSIX masochist (probably closer to 4N)

Looks like a lot of your scripts couldbe aliases, I store mine in ~/.aliases, and slightly more complicated things (e.g. take arguments) in ~/.functions, and source both those files from bashrc.

I have maybe 3 standalone shell scripts in my PATH, despite writing thousands of lines of shell.

Re: Writing Safe Shell Scripts (2019)

#33

Post by lottin »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

It's horses for courses. The Unix shell is an interface to system commands. Python is general purpose programming language.

Re: Writing Safe Shell Scripts (2019)

#34

Post by butterthebuddha »

aidenn0 wrote: post #6
mekster wrote: post #2

Make sure to have ShellCheck either integrated in your editor or run it before executing. It really tells many of the rules you're supposed to abide by and helps you write cleaner shell script. https://github.com/koalaman/shellcheck

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.

Re: Writing Safe Shell Scripts (2019)

#35

Post by _bxg1 »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

When you don't need to do any actual logic and just want to run a sequence of commands that you've been typing in by hand, and you already know the commands, and you don't want to go look up the Python versions of them all.

Re: Writing Safe Shell Scripts (2019)

#36

Post by ridiculous_fish »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

git extensions are an unfortunate counter-example. git relies heavily on shell scripts, and it even has a shell "library" [1].

I wanted to create a custom git command and started with Rust/libgit2, but found it was missing too much. Shell scripting proved to be the most natural, best supported approach (though it was nevertheless very painful).

1: https://github.com/git/git/blob/master/git-sh-setup.sh

2: https://github.com/ridiculousfish/git-prev-next

Re: Writing Safe Shell Scripts (2019)

#37

Post by chousuke »

aidenn0 wrote: post #14
chousuke wrote: post #5

Can mktemp fail? I recently wrote a script that ensures that the directory created by mktemp actually exists and starts with /tmp so that when the script wipes out its temporary data at the end, it will not ever run something like "rm -rf /". Using fully-qualified paths for everything is also (not) fun. In another script I was leery about running rm -f -- /path/foo* so I instead used find /path -mindepth 1 -maxdepth…

With the "--" and assuming GNU rm, rm -f -- /path/foo* is safe. If you use a glob pattern that does not match itself, then you will want nullglob: rm foo.[cC] will remove a file named literally foo.[Cc] if neither foo.c nor foo.C exist, despite the fact that foo.[Cc] does not match the glob. nullglob is probably a good default for bashscripts, but I don't think it exists in posix.

Wouldn't the glob still have issues with files that contain spaces?

My defensiveness generally takes a sharp upturn when I write scripts that will be run as root or ones that delete data, but I'll admit not trusting mktemp's exit code is probably taking it a bit too far.

Re: Writing Safe Shell Scripts (2019)

#38

Post by ghostwriter »

lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I used to do that with a help of Fabric/Invoke, and it was a pleasant experience overall, but sometimes required too much of verbosity. Then I discovered Plumbum (https://plumbum.readthedocs.io/en/latest/) with its concept of combinators, that I liked a lot more.

Eventually it motivated me to switch the language for shell scripting for the second time, and nowadays I recommend Haskell's Turtle (https://hackage.haskell.org/package/turtle-1.5.16/docs/Turtl...) to anyone who is interested in safer shell scripting and still likes a concise and terse syntax (and it's blazingly fast too).

Re: Writing Safe Shell Scripts (2019)

#39

Post by bobbyi_settv »

chousuke wrote: post #5

Can mktemp fail? I recently wrote a script that ensures that the directory created by mktemp actually exists and starts with /tmp so that when the script wipes out its temporary data at the end, it will not ever run something like "rm -rf /". Using fully-qualified paths for everything is also (not) fun. In another script I was leery about running rm -f -- /path/foo* so I instead used find /path -mindepth 1 -maxdepth…

The point of mktemp is to use the right location on the current system, which may not be /tmp. If you're going to enforce that tmpdir must be /tmp, you might as well not bother with mktemp and just directly create a file or directory in /tmp.

Re: Writing Safe Shell Scripts (2019)

#40

Post by ddebernardy »

sillysaurusx wrote: post #19
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

Ok, go ahead and port all 200 of these scripts to Python please: https://github.com/shawwn/scrap PR's welcome. ;) Python makes sense for scripts that need good argument parsing, or complicated intermediate input processing. But it's pretty annoying to get a shell pipeline working in Python. `foo | bar | baz` is about 15 characters in bash. Here's an example. I use `llbranch` all the time: https://github.com/shawwn/sc…

Correct me if my first impression was incorrect, but scanning through your scripts they all seemed like sub-kilobyte one-liners. Which, yeah, just use Shell for those and you'll be fine.

To my mind, the parent post was referring to lengthier, more complicated scripts. Speaking as one who wrote and routinely maintained such scripts at one point, I cannot agree more with the sentiment that most if not all of them could and should have been written in Python or Ruby or some other scripting language instead.

Post reply on HN