Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing Safe Shell Scripts (2019)

sipb.mit.edu

Post reply on HN
Print view View original

61–70 of 166 posts

Re: Writing Safe Shell Scripts (2019)

#61

Post by kohtatsu »

ben509 wrote: post #46
kohtatsu wrote: post #32

Earlier quoted context omitted.

Looks like a lot of your scripts couldbe aliases, I store mine in ~/.aliases, and slightly more complicated things (e.g. take arguments) in ~/.functions, and source both those files from bashrc. I have maybe 3 standalone shell scripts in my PATH, despite writing thousands of lines of shell.

I had a mess of shell functions. It is nice that they run more quickly, but I don't have any shell commands that need to run that often. What moved me to putting them in a bin directory was that it's just easier to edit a standalone script.

Write a shell script to concat them into a single file to source ;)

Re: Writing Safe Shell Scripts (2019)

#62

Post by edgyquant »

Pxtl wrote: post #58
heavyset_go wrote: post #30

Earlier quoted context omitted.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

PowerShell, mostly because PowerShell was designed as a mashup of Bash and C#.... And it's kind of a trainwreck in a lot of ways. It really feels like piping and easy process invocation and compile-time directory awareness wouldn't be massively onerous to add to an existing full-featured programming language so you wouldn't have to sacrifice a good type system and powerful syntax when you want to do scripty things.

Right. Powershell is a shell though not a programming language.

Re: Writing Safe Shell Scripts (2019)

#63

Post by seorphates »

Not to be that guy but this is mostly garbage. I suggest simply paying attention and testing. Switches can be a useful part of the tool but they can break things. It doesn't make things better rather it alters behaviors of the shell in sometimes unintended ways. Try some loops with some -eo. If you have a pipe that fails your script you're just ripping the heart and potential out of your script. I think you might just be looking for one-liners. It can be good to catch your failures and try other things vs "woop, damn, nope." Maybe you need things to fail first.

"Quote liberally" - also ripe. Mind your aPostrophes and Quotes because they do things.

I like writing for me and what I need done, not what or how others might have me write or how others might think it should be done.

The shell is a base orchestration and interface tool for your os. If you think it's just that bad then just stick to your language of choice but please refrain from suggesting lazy habits make a better shell, they don't.

Fly your own kite. It's much more fun.

Re: Writing Safe Shell Scripts (2019)

#64

Post by silly_giraffe »

chubot wrote: post #49
butterthebuddha wrote: post #34

Earlier quoted context omitted.

I really want someone to integrate fish and oil, and call the project "fish-oil" or something along those lines.

(Oil author here) Yes I think that's a great idea, and I hope to release a "liboil" so that people can do that. Let a thousand flowers bloom, etc. The most likely thing is that Oil is going to be a much better language this year (more expressive, with safety features, and it should be fast), but the interactive UI will still be more bash-like than fish-like: https://www.oilshell.org/blog/2020/01/making-plans.html So…

I want to say that I am continually impressed at your long-term vision and commitment you have taken to the Oil project. Your posts are a treat to read, and the care you have given to the problem is appreciated. We need to stop paving cow-paths, and discard historical baggage if we are ever going to build robust systems.

Re: Writing Safe Shell Scripts (2019)

#65

Post by IshKebab »

heavyset_go wrote: post #30
lalaland1125 wrote: post #15

I really wonder whether there is really any point in writing shell scripts anymore. Practically every Unix/Linux box in existence has at least some version of Python 2 that can be used as a complete and total replacement. I can't think of a single situation where I would need a shell script and a Python script wouldn't be much cleaner, simpler, and more maintainable.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

I don't know about process substitution, but Dart makes connecting processes together pretty easy and you can do way more fancy things than Bash let's you do, in a clean fast language with basically no gotchas.

Check out dshell: https://pub.dev/packages/dshell

Re: Writing Safe Shell Scripts (2019)

#66

Post by desc »

Pxtl wrote: post #54
pfranz wrote: post #26

Earlier quoted context omitted.

I've primarily used Python 2 for 10+ years and I often find cases where shell scripts are preferable. The major differentiator is usually "shelling out" in Python kind of sucks. It's verbose, output collection and error handling suck, and escaping can be miserable. I often will reimplement things in pure Python if I have the time. A recent example was I needed to tar+split large files. `tar cf - -C / $filename | spli…

Yeah, most of that comes from the verbose process needed to invoke a process, right? That's something I noticed when going back and forth between PowerShell and C# - that if C# had clean support for invoking a process and collecting the results as an IEnumerable like PowerShell does, PS wouldn't really need to exist, since 90% of the time you're dropping into C#/.net objects to get anything done anyways.

I actually invested some time a while back in building a nicer API for C# to invoke shell commands and process the results. The only downside IMO is the Rx library dependency for STDOUT/STDERR; I personally try to avoid depending on libraries which themselves have extra dependencies.

Since I did this at work it belongs to my employer, so I can't currently publish it freely, but it's not part of our core product so they may be open to publishing it under MIT licence or similar at some stage.

So it can be done, and has been done, but I guess most people are sufficiently happy with Powershell, Python, etc to not bother bridging the gap for C# too.

Re: Writing Safe Shell Scripts (2019)

#67

Post by heavyset_go »

IshKebab wrote: post #65
heavyset_go wrote: post #30

Earlier quoted context omitted.

I've yet to find a programming language that makes I/O redirection, piping, and process substitution[1] as easy as Bash does. Process substitution is where the shell really shines, in my opinion. Bash, and Bash-like shells, are literally everywhere. I have to be wary about what Python 3 features I use, and if there will even be a Python interpreter available. My OpenWRT router has a Bash shell, but I don't care to in…

I don't know about process substitution, but Dart makes connecting processes together pretty easy and you can do way more fancy things than Bash let's you do, in a clean fast language with basically no gotchas. Check out dshell: https://pub.dev/packages/dshell

Pretty cool, thanks for pointing it out. I have some experience with Dart via Flutter development, so I'll give this a go when I have some time.

Re: Writing Safe Shell Scripts (2019)

#69

Post by mehrdadn »

chubot wrote: post #52
aidenn0 wrote: post #6

Earlier quoted context omitted.

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

It's a burden to you only because you're used to not putting quotes in some cases. But to some of us who always put the quotes, it's not really about that at all, and it's not a subjective issue - missing them is literally wrong. It's like sprinkling random .split(" ") calls in your Python code for no reason, which you'd (hopefully?) never do. And when something is semantically so wrong, the elegance of its syntax compared to a correct solution becomes pretty much irrelevant.

Re: Writing Safe Shell Scripts (2019)

#70

Post by hkiely »

chubot wrote: post #52
aidenn0 wrote: post #6

Earlier quoted context omitted.

Shellcheck is amazingly impressive at catching issues with shell scripts. It makes it very hard to write a shell script that does the wrong thing. Also, look at oilshell[1]; it is bash compatible out-of-the-box, but has several options to make it incompatible, but safer (e.g. no field splitting of parameter expansion by default, making quotes much less needed). 1: https://www.oilshell.org/

FWIW I think ShellCheck is great and the state of the art, but Oil is partly (negatively) inspired by ShellCheck :) Somebody integrated ShellCheck into Google's code review system about four years ago, right before I left. So the result was that every code review I sent with a shell script was filled with red squigglies -- "add double quotes here". Most code reviewers don't really know shell, but if they see red squi…

Which is best for simply catching typing errors? Shell check? I never saw the instillation instructions for oil.
Post reply on HN