Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe ways to do things in bash

github.com

Post reply on HN
Print view View original

131–140 of 255 posts

Re: Safe ways to do things in bash

#131

Post by themodelplumber »

freedomben wrote: post #122
themodelplumber wrote: post #111

Earlier quoted context omitted.

> 6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing. Stories! C'mon, it was a long Monday :-)

Haha. Mostly it is dozens, probably hundreds of little things over time when turning to the shell proved highly productive and effective. The shell is surprisingly really, really good at processing text (using the good 'ol unix tools), and it's amazing how often "text processing" type problems come up. Whether you are doing find/replace on source code, looking for where certain strings are used/defined, or curling so…

That's really cool, thanks. It seems that whenever I come across an annoying text-processing problem, it's always really hard or too generic-sounding to get search engine help with potential bash solutions. However I'd somehow like to improve in that area.

Re: Safe ways to do things in bash

#132

Post by craftyguy »

koolba wrote: post #8

From the article: > Should I use curly braces? Bad: some_command $arg1 $arg2 $arg3 Extra bad (cargo culting unnecessary braces): some_command ${arg1} ${arg2} ${arg3} Correct: some_command "${arg1}" "${arg2}" "${arg3}" Better: some_command "$arg1" "$arg2" "$arg3" > In the "extra bad" and "correct" examples, braces compete with quotes under the limits of tolerable verbosity. > Shellharden will rewrite all these variant…

I agree. Consistency wins over typing 2 extra characters. It's much easier (for me at least) to scan shell scripts and notice the variables when they are written consistently.. and that means using braces.

Re: Safe ways to do things in bash

#134

Post by [deleted] »

freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

[deleted]

Re: Safe ways to do things in bash

#135

Post by tokyodude »

Xcelerate wrote: post #78

I always think — when a programming/scripting language requires this much bizarre knowledge just to write basic code that performs basic tasks, perhaps it is time for that language to be retired. I really don't understand why bash still exists. I've switched to fish and am much happier with the change.

I agree with you 100%

It basically comes down to tradition, ubiquitousness, and stubbornness. Which is sad to me. We fix other stuff. If an exploit appears in the OS it gets patched. Yet here we have a system that basically defaults to bad behavior, un-secure. It should be redesigned/replaced with something that makes it hard to do it wrong. If not replacing then maybe adding a "safe" mode and slowly deprecating the non-safe way. Sure it may take years. So did moving from python 2 to 3 but it is happening.

Maybe it will take the powers-that-be to start having their data stolen though shell exploits to realize if they made the shell hard to get wrong by default they'd be helping themselves as well?

Re: Safe ways to do things in bash

#136

Post by rdtsc »

freedomben wrote: post #121
antris wrote: post #113

Earlier quoted context omitted.

What is your opinion on using other languages to augment bash? Awk, perl etc.

I personally consider awk, sed, grep, cut, etc. to be "part" of bash. I don't think I ever write a script without invoking those venerable tools at least once. I often have many pipes, such as some_command \ | grep -E 'some.*regex$' \ | sed -e 's/erase_text//g' \ | sed -e 's/erase_more//g' \ | awk '{ print $2 }' \ | cut -d ':' -f 1 I used to reach for Perl one-liners a lot, and still do sometimes when I need a gross…

> sed -e 's/erase_text//g' \

I like the pattern. Copied it to my notes to try next time I am grepping through logs.

I usually just string a few 'grep -v ignore_this' with pipes but then it ignores the whole line. But I can see how erasing some parts of the line would be very helpful sometimes.

Re: Safe ways to do things in bash

#137

Post by Annatar »

" Should I use backticks?

Command substitutions also come in this form:

    Correct: "`cmd`"
    Bad: `cmd`
While it is possible to use this style correctly, it looks even more awkward in quotes and is less readable when nested. The consensus around this one is pretty clear: Avoid."

This is how one can tell a rookie just learning to program in shell: usage of $() syntax is limited to Bourne family of shells which implement that particular POSIX specification aspect.

Backticks, on the other hand, although they incur a performance penalty since they spawn a subshell, make one's code instantly portable across all UNIX-like operating systems and even across disparate shell families, as they work exactly the same in C-shells. (Whether one should program in a C-shell family is a different discussion.)

The subshell performance penalty is negligible in 99% of the cases as this 1970's technology has tiny memory and processor overhead due to the fact that it's been developed on systems with small memory and a slow CPU, so it had been optimized for performance.

Over my 30+ years of shell programming, I know of only one documented instance where the $() construct which doesn't spawn a subshell made a difference, and was the only time it was actually a valid requirement:

https://www.joyent.com/blog/building-packages-at-scale

but even then, the author ended up using dash, not bash.

For maximum portability and closest adherence to POSIX, program in Korn shell, ksh93. (Modern versions of ksh implement ksh93 functionality.) Then you may safely use $() and be assured it will work in all Korn shells across different operating systems (even in ksh88).

Otherwise, DON'T avoid using backticks, because you will be giving away portability for no good reason. Don't program in bash, but in original Bourne shell (sh) for maximum portability across different operating systems; don't assume that you can use bash constructs in Bourne shell (as /bin/sh on GNU/Linux tells bash to run in Bourne shell emulation mode, but that mode isn't implemented completely or correctly, since bash constructs are still accepted). Always test your shell code on a traditional UNIX like HP-UX or a Solaris derivative like SmartOS if possible, with a real Bourne shell.

Re: Safe ways to do things in bash

#138

Post by mihaifm »

LeoPanthera wrote: post #80
Xcelerate wrote: post #78

I always think — when a programming/scripting language requires this much bizarre knowledge just to write basic code that performs basic tasks, perhaps it is time for that language to be retired. I really don't understand why bash still exists. I've switched to fish and am much happier with the change.

Because it is ubiquitous. You can virtually guarantee that bash will be found on any arbitrary unix-like system.

Don't make that assumption. I worked on older Solaris installations, they did have bash but the production scripts were written in ksh 88. I had to dig up obscure features of this shell in because they didn't even upgrade to the newer version ksh 93 (yes, those numbers come from their release dates...1988 and 1993).

Re: Safe ways to do things in bash

#139

Post by laumars »

themodelplumber wrote: post #111
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

> 6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing. Stories! C'mon, it was a long Monday :-)

Back before DevOps was a thing, us sysadmins were building the same automation tools in shell scripts. Using rsync, git (cli) and such like instead of docker, GitHub plugins, concourse / AWS Code Deploy etc. And ssh keys on VMware / xen stacks instead of puppet / terraform etc.

As much as I think some of the new generation of tools are pretty awesome for getting stuff done, some of it honestly feels like a step backwards because you're relying on 3rd party services or require a new server(s) to be provisioned to host your management servers; some of which require extra man power just to keep running. And as much as Bash gets a bad rep for hidden traps, I'd take that any day over Terraforms "I'm not really a programming language but I like to pretend to be" markup, HCL.

But that's the nature of the game now; it's less about hacking stuff together and more about using preformed Lego bricks to build the same approximation.

Re: Safe ways to do things in bash

#140

Post by Groxx »

freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

Re 4: bash is an acceptably-good domain-specific language.

Which leads right into 5: that domain is unsafe-by-design and failure-tolerant. Which is phenomenally useful and efficient for one-off tasks, and phenomenally dangerous for something you allow others to inject behavior into.[1]

There are a fair number of things which bash could do better, and probably should. But many of the gotchas like "if you don't quote it, it'll expand into multiple arguments" are features when you want to throw things around efficiently. It wouldn't be as efficient for plugging so many disparate tools together if it weren't such an efficient footgun.

[1]: If you need anything even remotely fault-resistant, `set -euo pipefail` and use `shellcheck` (and the rest of this resource, it is indeed great) and then consider using a "normal" programming language instead. But for behavior you fully control it's not too bad if you're careful.

Post reply on HN