Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe ways to do things in bash

github.com

Post reply on HN
Print view View original

221–230 of 255 posts

Re: Safe ways to do things in bash

#221

Post by robot01 »

opinionator1 wrote: post #200
znpy wrote: post #192

Earlier quoted context omitted.

> So, you'd rather use something that is decades old because you can't/don't want to (ask to) install 1 new program on the server and take a weekend to rebuild your config file? Absolutely, yes. Because long story short, bash is TriedAndTrue® technology. Stuff that works. It takes a bit of dedication to be mastered at a decent level but it pays off immensely. It's so ubiquitous it is one of those tools that you can l…

"works" More like "better the devil you know" (if you can even say that much). Bash doesn't scale. Every shop larger than 1 has been burned by bash gotchas. Use a real scripting language and shell out to the commands and builtins when necessary. At my shop bash is strictly disallowed in production environments and we're all better off for it.

Do you guys have a different, approved shell for prod?

Re: Safe ways to do things in bash

#222

Post by znpy »

opinionator1 wrote: post #200
znpy wrote: post #192

Earlier quoted context omitted.

> So, you'd rather use something that is decades old because you can't/don't want to (ask to) install 1 new program on the server and take a weekend to rebuild your config file? Absolutely, yes. Because long story short, bash is TriedAndTrue® technology. Stuff that works. It takes a bit of dedication to be mastered at a decent level but it pays off immensely. It's so ubiquitous it is one of those tools that you can l…

"works" More like "better the devil you know" (if you can even say that much). Bash doesn't scale. Every shop larger than 1 has been burned by bash gotchas. Use a real scripting language and shell out to the commands and builtins when necessary. At my shop bash is strictly disallowed in production environments and we're all better off for it.

> Use a real scripting language and shell out to the commands and builtins when necessary.

Not to mention, for example: https://julialang.org/blog/2012/03/shelling-out-sucks

Re: Safe ways to do things in bash

#223

Post by rcxdude »

yAnonymous wrote: post #179
emmelaich wrote: post #147

Earlier quoted context omitted.

I probably have a similar history but I've come to the belief that you should not use Bash if you're doing anything fancy or long (>10 lines). Just use Perl, Python or Ruby. One or all of them is installed on every machine you are likely to use.

Is there an easy way to run shell commands in Python, like backticks in PERL, Ruby or even PHP? I'd really like to use Python more often, but most of the time it only complicates things with those verbose syscalls.

part of the complication is due to actually having the proper data seperation that you need to jump through hoops to get in bash. If you go with backtick style calls you lose a lot of that.

Re: Safe ways to do things in bash

#224

Post by laumars »

opinionator1 wrote: post #202
laumars wrote: post #139

Earlier quoted context omitted.

Back before DevOps was a thing, us sysadmins were building the same automation tools in shell scripts. Using rsync, git (cli) and such like instead of docker, GitHub plugins, concourse / AWS Code Deploy etc. And ssh keys on VMware / xen stacks instead of puppet / terraform etc. As much as I think some of the new generation of tools are pretty awesome for getting stuff done, some of it honestly feels like a step backw…

You need to fix your mentality. HCL not being a complete language is a feature. Bash is a liability. If you honestly think you can build more reliable modern systems in bash, you deserve to be laughed right back into the 70s where you belong.

I rarely bother responding to troll accounts but I do feel the need to point a couple of things out:

1/ HCL not being a complete language is a burden when you're writing build scripts that need to target multiple environments (eg dev, UAT, staging, live) as it means you have to use the `count` syntax to include/exclude resources. Also I've frequently run into issues where output will expect a variable or resource to exist even when it's inside a ternary operator where the condition isn't met (isn't the point of an `if` condition that you don't evaluate all blocks of code in the structure?).

2/ The reason HCL isn't a "complete language" isn't because "complete languages" are a liability (clearly that's nonsense); it's because Hashicorp wanted something that met the middle ground between JSON and scripting so that all ranges of technical expertese were comfortable; and they wanted a format that could still be compiled back down to JSON. Personally I feel the monster they created is the worst of both worlds but I do think the goal they were trying to achieve is an honorable one.

3/ I was never trying to make the point that I've built more reliable systems in Bash. However once you start writing pretty complex Terraform projects (some of my build scripts are a several thousand lines of code; inc modules) you really do start to bang your head with Terraform - not all of it because of HCL though. Don't get me wrong, Terraform is definitely the best tool we have at the moment for deploying to "the cloud"; but that doesn't mean it isn't still bloody annoying to work with at times. A few bugs I've run into in the last month:

3a/ Error message wasn't being passed from AWS to the user so infra would fail to build but without explanation. A few hours of debugging and we discovered what resource was failing and why. This was a bug in Terraform so issue raised on Github.

3b/ Workspace name pushed the resource name tag over it's character limit. We did get an error message here but it was inaccurate. Thankfully this is an issue we've run into before so it was pretty quick to resolve.

3c/ The aforementioned problem of all code getting evaluated inside both conditions of a ternary operator

3d/ If your AWS token expires before the Terraform apply is complete you're left with incomplete infrastructure and no valid state file to rollback. This is particularly annoying if your builds do certificate management or ELK; both of which can take 15 to 20 minutes just on those resources alone.

3e/ You don't get line numbers nor even file names when errors are raised which makes debugging through large projects painful.

4/ I'm not about to sing the praises of Bash as a modern deployment tool as it can be a complete nightmare to work with if you're not proficient in it's hidden traps. But to answer your question; yes I have built reliable systems in Bash. It's a tool that has been around for decades, people can and have built some pretty reliable stuff in it and I personally consider myself a veteran. (for what it's worth I have nearly 3 decades of dev skills and have used a lot of other languages we might now consider "dangerous" - some of which are still used in production). As an aside note; it's the decades of years of experience in Bash and other, much lower level, languages that inspired me to write my own $SHELL and scripting language. So I tend to use that more than Bash these days.

5/ I didn't proof read this so apologies if some parts didn't make any sense. It's a long post and in replying to you I'm interrupting fixing one of those massive Terraform projects I described earlier.

Re: Safe ways to do things in bash

#225

Post by JdeBP »

h1d wrote: post #155
zwischenzug wrote: post #127

Earlier quoted context omitted.

I did a talk on this a while ago: https://www.youtube.com/watch?v=pb3k0sGKrjQ&t=457s 'Take bash seriously'

It's time we need a better shell than bash instead of thinking it's great after bleeding with it for years and gets used to it.

We have just had the 30th anniversary of someone acting upon that very thought.

> Perl is a interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information. It's also a good language for many system management tasks. The language is intended to be practical (easy to use, efficient, complete) rather than beautiful (tiny, elegant, minimal). It combines (in the author's opinion, anyway) some of the best features of C, sed, awk, and sh, so people familiar with those languages should have little difficulty with it. (Language historians will also note some vestiges of csh, Pascal, and even BASIC-PLUS.) Expression syntax corresponds quite closely to C expression syntax. If you have a problem that would ordinarily use sed or awk or sh, but it exceeds their capabilities or must run a little faster, and you don't want to write the silly thing in C, then perl may be for you. There are also translators to turn your sed and awk scripts into perl scripts.

-- Larry Wall, perl 1.0 release announcement, comp.sources.unix v13i001, 1988-01-02.

Re: Safe ways to do things in bash

#227

Post by swinglock »

geofft wrote: post #129
foxylad wrote: post #126

Earlier quoted context omitted.

> without excessive verbosity If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

By "excessive verbosity" I do mean poor readability and poor maintainability. If you're invoking 5 commands and most of your Python code is wiring up the commands to each other right, just write 5 lines of shell, don't make me pull up the subprocess docs to see if your 30 lines of Python are doing the same thing and how to make a small change without risking pipes deadlocking. Python is readable and maintainable when…

Does the Plumbum library grant your wishes? I've not used it myself so I don't know, but its purpose is to make you "never write shell scripts again", so it might.

Re: Safe ways to do things in bash

#228

Post by petee »

h1d wrote: post #213
petee wrote: post #187

Earlier quoted context omitted.

Avoiding the next Shellshock? As bad as it was, who knows what goodies an even less-audited shell has in store

Using bash never prevented the problem in the first place.

Bash was the problem

https://en.wikipedia.org/wiki/Shellshock_(software_bug)

Re: Safe ways to do things in bash

#229

Post by mncolinlee »

davedx wrote: post #152
emmelaich wrote: post #147

Earlier quoted context omitted.

I probably have a similar history but I've come to the belief that you should not use Bash if you're doing anything fancy or long (>10 lines). Just use Perl, Python or Ruby. One or all of them is installed on every machine you are likely to use.

Yes, but which one, and which version? This is especially painful with Python ;)

KotlinScript is yet another option that handles that worry nicely.

https://github.com/holgerbrandl/kscript

Re: Safe ways to do things in bash

#230

Post by wheaties »

h1d wrote: post #153
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

I think it's time people start using something better than bash/zsh that is decades old, like fish or even come up with a more modern shell. Even by looking at these examples, you see it has less verbosity like "then" and "do", you can reference arguments as $argv instead of cryptic $@ and exit status code as $status instead of $? which is confusing with $! and the likes. https://blog.codeship.com/lets-talk-about-she…

What's wrong with something that's decades old? Bash is great for command line and small scripts. It's once you surpass 100 lines where things get problematic. At that point, you're not writing a shell script, you're writing a tiny application and need to treat it as such.
Post reply on HN