Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

141–150 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#141

Post by effie »

zelly wrote: post #96
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

The shell is the true user interface to the OS. I like to think of writing shell scripts as having some mouse clicking automation tool on a GUI shell. It should be viewed as just as kludgy of a solution. But there really isn't anything more convenient most of the time. Shell scripts are to be called by humans to automate behavior they otherwise would have had to do manually. An application should never rely on a shel…

> An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix).

Could you explain why? I'm very happy with programs calling shell commands/scripts. You execute another program the same way whether interactively from a shell, or by calling system() from another program. The simplicity and universality of the call syntax is an advantage.

Re: Writing Safe Shell Scripts (2019)

#142

Post by derefr »

I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution?

Hopefully not by generating a megabyte of polyfill runtime code; more just by the compiler refusing to compile code in $lang unless it has a direct equivalent in /bin/sh. Any $lang source file the compiler accepted, would just look like “a shell script translated into $lang” already. But the compiler would inject $lang’s safe semantics (exceptions, type-checking, etc.) during the compilation, so you’d at least get that benefit.

Alternately, I’d also be satisfied with “emscripten for shell”: a compiler that generated shell scripts containing a small WASM-like emulator and an embedded bytecode stream to feed it. As long as it was lightweight enough. (A large point of these environments’ use of /bin/sh is that they’re small and embedded and can’t load too much into memory at once.)

My only guess for why this hasn’t happened, is that the people who write things like shell scripts that execute in initramfs, or shell scripts that are intended to install stuff even on lesser-known UNIXes, are all old-hand ops people who know shell-scripting cold, and certainly aren’t developers with any experience in compiler theory.

Re: Writing Safe Shell Scripts (2019)

#143

Post by cdaringe »

derefr wrote: post #142

I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution? Hopefully not by generating a megabyte of polyfill runtime code; more…

Because doing so would be hard for shell scripts to express. real functions, try/catch, and other common language features are not often part of shell languages. As such, the generated code may not support many features you typed in and/or the generated code would bundle a runtime with it to emulate what we'd otherwise consider pragmatic scripting feature support.

Re: Writing Safe Shell Scripts (2019)

#144

Post by derefr »

cdaringe wrote: post #143
derefr wrote: post #142

I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution? Hopefully not by generating a megabyte of polyfill runtime code; more…

Because doing so would be hard for shell scripts to express. real functions, try/catch, and other common language features are not often part of shell languages. As such, the generated code may not support many features you typed in and/or the generated code would bundle a runtime with it to emulate what we'd otherwise consider pragmatic scripting feature support.

I mean, like I said, I don’t want to use $lang features that don’t exist in shell script. E.g., I want “exceptions” in the sense that adding a string-typed variable to an integer-typed variable will blow up in a descriptive way (or better, not compile); but I don’t want real exceptions in the sense of being able to catch them. I just want everything to translate to the shell script aborting in verbose and helpful ways before doing something stupid with invalid inputs; or not compiling at all if there’s a code-path that can’t possibly be valid.

Also, I wouldn’t mind if this $lang that compiles to shell script is its very own language I’d have to learn, just like TypeScript is its very own language you have to learn. As long as I don’t have to manually write five layers of guards using impenetrable [ “y${x:-1}” -eq “f” ] style code, and then duplicate the hierarchy of cleanup behaviours after each failed guard, I’d be happy.

Re: Writing Safe Shell Scripts (2019)

#145

Post by mehrdadn »

effie wrote: post #138
mehrdadn wrote: post #69

Earlier quoted context omitted.

It's a burden to you only because you're used to not putting quotes in some cases. But to some of us who always put the quotes, it's not really about that at all, and it's not a subjective issue - missing them is literally wrong . It's like sprinkling random .split(" ") calls in your Python code for no reason, which you'd (hopefully?) never do. And when something is semantically so wrong, the elegance of its syntax c…

Quotes are just a tool, not a religious requisite. Chubot is talking from experience, and he is right. Putting quotes on all shell expressions is a burden, because it's more keystrokes, and more characters on the display which make the source on display disorienting. When you use shell every day, you know where the quotes are needed and where they are just a noise. Use a tool - when it's needed.

Yes, of course they're tools, but "religious requisite" is a pretty disingenuous representation of the argument I'm making here? Something being a tool doesn't mean its use or lack thereof can't be incorrect. Bikes are tools too, but you don't ride them on the highway, right?

Quotes are tools that affect the correctness of your control flow quite vividly. Of course if there are multiple tools for solving the correctness issue you're welcome to choose any of them (and maybe the choice between the tools is something you can ascribe to religion), but that doesn't absolve you of the need to solve the issue itself. When the argument is that a piece of code is executing objectively wrong instructions (and ones that would be so obviously wrong in other languages!), that's pretty much the diametrical opposite of what you can casually dismiss as religious dogma...

Re: Writing Safe Shell Scripts (2019)

#146

Post by zelly »

effie wrote: post #141
zelly wrote: post #96

Earlier quoted context omitted.

The shell is the true user interface to the OS. I like to think of writing shell scripts as having some mouse clicking automation tool on a GUI shell. It should be viewed as just as kludgy of a solution. But there really isn't anything more convenient most of the time. Shell scripts are to be called by humans to automate behavior they otherwise would have had to do manually. An application should never rely on a shel…

> An application should never rely on a shell script. If you have to execute another program, use the syscalls (fork and exec on posix). Could you explain why? I'm very happy with programs calling shell commands/scripts. You execute another program the same way whether interactively from a shell, or by calling system() from another program. The simplicity and universality of the call syntax is an advantage.

Security. system() is one of the most common targets for hacking (getting a shell by manipulating the string passed to system() by various means). Calling programs from the kernel directly is a lot more well-behaved. You're limited to only executing one program.

Re: Writing Safe Shell Scripts (2019)

#147

Post by chubot »

fctorial wrote: post #108
chubot wrote: post #79

Earlier quoted context omitted.

Yeah I don't disagree with that. If you're working in a big group with some shell scripts, and want to get stuff done, use ShellCheck. I just find it annoying, hence the new shell :) There's a lot more to Oil, but that's definitely one of the surface annoyances I want to fix. Although zsh actually does fix that, and for some reason I've almost never seen a zsh script.

Many oh-my-zsh plugins are just collection of functions, which I think could be considered scripts. https://github.com/ohmyzsh/ohmyzsh/wiki/Plugins

Yes sorry I should have said "I rarely see a batch program in zsh". zsh is customized with zsh scripts for sure, and they are very extensive.

I guess it's because zsh is not POSIX compatible by default.

Re: Writing Safe Shell Scripts (2019)

#148

Post by limeblack »

derefr wrote: post #142

I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution? Hopefully not by generating a megabyte of polyfill runtime code; more…

This does exist but bash/sh is missing proper types. Examples include, null and numbers and anonymous functions.

Mathematica and other software platforms use bash as universal installer for Mac and Linux. I have seen languages that compile to bash but use is limited based on the language.

Re: Writing Safe Shell Scripts (2019)

#149

Post by ktpsns »

limeblack wrote: post #148
derefr wrote: post #142

I’m constantly surprised that we got a safe language that compiles to Javascript (TypeScript) but never got a safe language that compiles to shell script. Why can’t I write in (a subset of) some other scripting language $lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile it to actual portable shell script for distribution? Hopefully not by generating a megabyte of polyfill runtime code; more…

This does exist but bash/sh is missing proper types. Examples include, null and numbers and anonymous functions. Mathematica and other software platforms use bash as universal installer for Mac and Linux. I have seen languages that compile to bash but use is limited based on the language.

In many (not all) times, these "universal installers" are only a small number of POSIX-compilant shell lines followed by some Base64-encoded binary. The whole purpose of the shell commands is to extract this binary.

Re: Writing Safe Shell Scripts (2019)

#150

Post by boring_twenties »

heavyset_go wrote: post #30
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

If your OpenWRT router has a bash shell, then you optionally installed it from packages. The default shell is Busybox, which is a minimal shell that supports few if any "bashisms." In other words, you certainly do have to be wary about what shell features you use.
Post reply on HN