Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to do things safely in Bash (2018)

github.com

Post reply on HN
Print view View original

61–70 of 98 posts

Re: How to do things safely in Bash (2018)

#61

Post by pmiller2 »

stormbrew wrote: post #54
kubanczyk wrote: post #40

Earlier quoted context omitted.

Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you t…

It's worth noting that there are still a shocking number of production machines in the world that don't have stable installations of python3.x yet.

So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.

Re: How to do things safely in Bash (2018)

#62

Post by agumonkey »

belter wrote: post #33

Every time I think of writing some Bash I spend some time here: https://mywiki.wooledge.org/BashPitfalls and then ...I use something else.

Yeah it's ironic that you have such a wonderful repository of knowledge only to realize that you shouldn't have that much corner cases and bag of tricks.

Re: How to do things safely in Bash (2018)

#63

Post by marcosdumay »

rectang wrote: post #57
marcosdumay wrote: post #51

Earlier quoted context omitted.

Oh, I'm really not comparing it with Rust. The comparison is between a general language and a shell. If it's Python, Rust, PHP, Java, or whatever is of secondary relevance. I've done some ops work in C, Python, and C#. I'm also waiting for a good problem to try some Haskell shell monad. All the same old language pros and cons apply.

I'm just making a general argument about what "simple" means. A well-specified program may or may not be superficially simpler, but it will exhibit simpler behavior.

Oh, ok. The Rust version is at least as simple as the Python one, depending on what kinds of detail you care about.

The thing about shell code is that there are plenty of cases where you don't care about things like error management, bad inputs, and etc. So it's not immediately obvious what language is simpler. When you care about those, the shell becomes the most complex choice by far, and that's the duality problem that generates all those different opinions you see around.

Re: How to do things safely in Bash (2018)

#64

Post by JNRowe »

chubot wrote: post #10
nas wrote: post #6

The fact that you have to use quoting nearly everywhere is a design flaw in the Borne shell. Some shells, like Plan 9's rc, for example, don't expand after variable substantiation. They have an operator to call if you want to explicitly force expansion. That's so much cleaner and less error prone.

Oil is Bourne compatible, but has a mode to opt you into the better behavior. Example: osh$ empty='' osh$ x='name with spaces.mp3' This is like Bourne shell: $ argv $empty $x ['name', 'with', 'spaces.mp3'] # omit empty and split $ argv "$empty" "$x" ['', 'name with spaces.mp3'] # unchanged Opt into better behavior, also available with bin/oil: $ shopt --set oil:basic $ argv $empty $x ['', 'name with spaces.mp3'] # no…

> Interestingly zsh also doesn't split words

Should zsh users ever want that behavior they can enable it globally with the SH_WORD_SPLIT option, or more sensibly local to a given parameter with the = parameter expansion as in ${=var}. Also, it has the best comment in zsh's manpage: "SH_WORD_SPLIT [...] Note that this option has nothing to do with word splitting."

    $ x='name with spaces.mp3'
    $ print -l -- $=x  # "print -l" displays one element per line
    name
    with
    spaces.mp3
> it silently removes empty strings

You can keep the empty strings too if needed, but it requires using the @ expansion flag as in ${(@)arr}. It all becomes superbly readable, here I'll prove it:

    $ arr=($x '' 'old file')
    $ print -l -- "${(@D)arr:A:gs/old/new/}"
    ~/Desktop/name with spaces.mp3

    ~/Desktop/new file
In all seriousness, the zsh default handling feels right for interactive usage in this case. I'd love the strictness of oil, so that I simply don't need to remember these things. Not yet quite ready to give up on the zshexpn(1) goodies though.

Re: How to do things safely in Bash (2018)

#65

Post by imiric »

jrockway wrote: post #26
pwdisswordfish8 wrote: post #2

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for e…

Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". Th…

I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks.

> I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set".

Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls.

> I find myself reaching for purpose-built tools rather than ad-hoc pipelines

It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell.

> I am getting more and more frustrated at basic shell mechanics like history handling

That's mostly a solved problem in many shells. Zsh supports it natively[2].

I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost.

[1]: http://redsymbol.net/articles/unofficial-bash-strict-mode/

[2]: https://nuclearsquid.com/writings/shared-history-in-zsh/

Re: How to do things safely in Bash (2018)

#66

Post by JNRowe »

tediousdemise wrote: post #55

PowerShell is cross platform now and it’s quite good. I would encourage people who’ve never tried it to give it a shot. Like 99% of other people who never used PoSh, I thought it was just a Windows shovelware replacement for Command Prompt. I was mistaken—the syntax is actually a lot simpler than bash, but it’s just as capable. One of the cooler features about PoSh is that you can leverage Visual Basic/C#/.NET from i…

I'd probably like to see powershell usage increase too, but I think we may be in the minority. Every time it comes up I check to see if it is available in Debian and see that the packaging bug¹ has been open for years without anyone caring enough to move on it. The referenced upstream bug² points out that it isn't capital-F Free enough to distribute [yet], but you'd expect more comments in the bug if people were pushing(or a package for the non-free section).

¹ https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834756

² https://github.com/PowerShell/PowerShell/issues/5869

Re: How to do things safely in Bash (2018)

#67

Post by imiric »

fillest wrote: post #43
kubanczyk wrote: post #40

Earlier quoted context omitted.

Here is a novel idea. I am allowed to call 'mv' or 'cp' binaries from bash. I am also allowed to call 'mv' or 'cp' binaries from python with subprocess! (Edit: bad example binaries, 'df' or 'tar' would be probably better.) I must choose not to install bash libraries for a bash script beyond builtins. I can choose not to install python libraries beyond builtins. It becomes a really simple tradeoff. Nobody forces you t…

It is OK to call whatever one needs actually - just minimize the usage of Bash language features (Python provides most of them). It is not complicated and requires no dependencies, here's a self-contained helper: https://gist.github.com/fillest/8d64f8fa0cdb1745bfc9c683cf39...

Great, would that work with pipes and redirections too? At that point you might as well use sh[1] which is a more advanced abstraction. My issue with these approaches is that you'll inevitably hit a bug or limitation where just using a shell language would've been easier to maintain and more portable.

Python has its uses, but replacing shell scripts is not one of them IMO. It it was then the problem didn't need to be a shell script to begin with (doesn't shell out often, needs complex data structures, etc.).

[1]: https://amoffat.github.io/sh/

Re: How to do things safely in Bash (2018)

#68

Post by codethief »

jrockway wrote: post #26
pwdisswordfish8 wrote: post #2

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for e…

Same. I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set". Stack Overflow always has 100 answers on how to do this, and every one has someone replying "well that doesn't work if..." In general, after 20 years of using Unix, I'm starting to sour on the "everything is a string" and "streams are just newline separated plain-text records". Th…

I couldn't agree more!

> I feel like if the Unix shell needed to be fundamentally reimagined, someone would have already done it. But they haven't.

AFAICT this has a lot to do with the fact that most people who'd be able to "reimagine" / fundamentally improve the Unix shell are also the people that are so intimately familiar with the shell and all its quirks that they don't see any need for improvement. It has always been that way, so it must stay that way.

> I feel like I'm going to have to fix that myself in the next couple years...

Please let me know when you do. :)

Re: How to do things safely in Bash (2018)

#70

Post by ajross »

rectang wrote: post #47
marcosdumay wrote: post #30

Earlier quoted context omitted.

Hum, no those people are probably writing the same kind of script you do. Their Python versions are indeed longer and more verbose. I always recommend Python anyway if you care about edge cases (what is not always). That longer script handles them on the obvious way, while the short and more readable shell script does something absolutely crazy every time a detail is different from planed. And if you try to correctly…

Which of these is simpler? Rust: fn add(a: i32, b: i32) -> i32 { a + b } Python: def add(a, b): return a + b Arguably the Python function is "simpler" because it occupies fewer characters. And yet, the Rust function is more tightly specified; it does less. It can't throw exceptions, unlike the Python function. Invalid programs where you pass the wrong type to the Rust function won't even compile, whereas the Python f…

I'm a little lost... neither of those functions is doing something you would be expected to do in a shell script.

The upthread point was that shell scripts are for coordinating the actions of separate programs run via their command lines and simple IPC like pipes. If that's not what you want to do, shell is the wrong language. If that *is* what you want to do, then you'll find Python and Rust are pretty severely handicapped.

Post reply on HN