Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe ways to do things in bash

github.com

Post reply on HN
Print view View original

211–220 of 255 posts

Re: Safe ways to do things in bash

#211

Post by h1d »

znpy wrote: post #195
h1d wrote: post #158

Earlier quoted context omitted.

I keep seeing this statement but this absolutely does not apply to everybody. How often can you just install fish or use other programming language instead of being forced to use bash? It's sad people keep using the default just because of being afraid that the next system you touch might not have it and you waste tremendous amount of productivity without using something better. I know a guy who used vim with default…

No one ever thinks of interoperability between people . Bash is known, to a variable depth, by pretty much every sysadmin (and a big part of being a sysadmin is being able to write shell scripts). So I might prefer the fish, and who came before me might have had a preference for zsh. So now I have to deal with three different shells: bash (default), fish (for the scripts i'll be writing from now on) and zsh (because…

Why can you not modify the editor environment that you use often?

Hard to see I need to deal with environments where I can do nothing but keep the default every time I use it.

Once you get the speed that is tuned to your liking, the default sounds like you're walking with legs strapped.

Re: Safe ways to do things in bash

#212

Post by meow_mix »

scbrg wrote: post #160
meow_mix wrote: post #133

Why ever resort to shell scripts when we have languages like ruby / python for anything more complex than installs? Clarity is king people

Because foo | awk '/bar/ {print $3}' is more clear than import subprocess foo = subprocess.Popen(['foo'], stdout=subprocess.PIPE) for line in foo.stdout: if 'bar' in line: try: print(line.split()[2]) except IndexError: print('') Sometimes shell scripts are more clear. Especially for tasks that involve running lots of external commands.

Nobody doubts that pipes and languages like awk are great for one liners, but I think that's a little besides the point of this post, which is advocating for things like the use of bash arrays:

``` array=( a b ) array+=(c) if [ ${#array[@]} -gt 0 ]; then rm -- "${array[@]}" fi ```

``` array = [a, b] array 0 ```

There's also nobody stopping you from using text processing tools like awk and sed, or bash one liners in ruby/python either, but I think we should leave the logic and arrays for scripting languages, no?

Re: Safe ways to do things in bash

#213

Post by h1d »

petee wrote: post #187
h1d wrote: post #173

Earlier quoted context omitted.

If you're convinced your productivity may change by ditching bash, I'm not sure what kind of counterargument the sysadmin will bring.

Avoiding the next Shellshock? As bad as it was, who knows what goodies an even less-audited shell has in store

Using bash never prevented the problem in the first place.

Re: Safe ways to do things in bash

#214

Post by [deleted] »

JdeBP wrote: post #197
jwilk wrote: post #149

Earlier quoted context omitted.

I'm all for "command -v", but do you know which OSes don't have "which" by default?

A straw man question. That's not the problem with which. * https://github.com/koalaman/shellcheck/wiki/SC2230 * https://unix.stackexchange.com/questions/85249/

[deleted]

Re: Safe ways to do things in bash

#215

Post by znpy »

h1d wrote: post #211
znpy wrote: post #195

Earlier quoted context omitted.

No one ever thinks of interoperability between people . Bash is known, to a variable depth, by pretty much every sysadmin (and a big part of being a sysadmin is being able to write shell scripts). So I might prefer the fish, and who came before me might have had a preference for zsh. So now I have to deal with three different shells: bash (default), fish (for the scripts i'll be writing from now on) and zsh (because…

Why can you not modify the editor environment that you use often? Hard to see I need to deal with environments where I can do nothing but keep the default every time I use it. Once you get the speed that is tuned to your liking, the default sounds like you're walking with legs strapped.

> Why can you not modify the editor environment that you use often?

Because I am not the only one accessing those environments (please note the plural here).

Currently there are about 13 other sysadmins in my team and we manage clients infrastructures among other things (managed services). From time to time someone from another team accesses those environment (not a sysadmin, but still familiar with the bash shell). Sometimes the client accesses those systems (rare, we try to discourage and avoid that).

Can you even imagine what a mess it would be if we all started applying our own favorite settings ?

(edit: fix grammar)

Re: Safe ways to do things in bash

#216

Post by rkeene2 »

dredmorbius wrote: post #130
rkeene2 wrote: post #112

Earlier quoted context omitted.

It now also does rewriting on the fly for certain shebang lines, as part of its C library, so unmodified scripts can run.

I didn't know that, thanks.

No problem. I wrote it because I version control my scripts and didn't want to fork them for running them on my laptop (Chromebook with Termux).

Re: Safe ways to do things in bash

#217

Post by saltcured »

digitalsushi wrote: post #90
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

Re 6: I am an /extremely/ mediocre developer but I have crafted some real 99th percentile bash skills (within my company, not globally) and I completely self-sustain myself on just that. (You're wondering how many people write bash where I am, and the answer is somewhere between 8 and 15 thousand people). It's funny how many things that seem kind of incredible for a pure bash solution, looking back over the past 21 y…

Over 20 years ago, I was home on a sick day in the middle of refactoring a build system based on Bourne shell. We could not use any of the new features of BASH but had to stick to what worked as /bin/sh across a bunch of supercomputer vendors. I was finding a lot of arcane tricks to optimize execution on those systems, some of which had improbably expensive fork-exec.

In my fevered state, I wrote some arithmetic functions to do the equivalent of `expr $a + $b` and `expr $a \* $b` without any fork-exec to external commands, just to prove to myself that it could be done. I think I abused things like $IFS, for loops, recursive function calls, and case statements to make an arbitrary-precision decimal adder function.

Re: Safe ways to do things in bash

#218

Post by stronglikedan »

h1d wrote: post #153
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

I think it's time people start using something better than bash/zsh that is decades old, like fish or even come up with a more modern shell. Even by looking at these examples, you see it has less verbosity like "then" and "do", you can reference arguments as $argv instead of cryptic $@ and exit status code as $status instead of $? which is confusing with $! and the likes. https://blog.codeship.com/lets-talk-about-she…

Why not both?

> Bash has arrays and a safe mode, which may make it just about acceptable under safe coding practices, when used correctly.

> Fish is easier to use correctly, but lacks a safe mode.

> Prototyping in fish is therefore a good idea, provided that you know how to translate correctly from fish to bash.

Re: Safe ways to do things in bash

#219

Post by znpy »

opinionator1 wrote: post #200
znpy wrote: post #192

Earlier quoted context omitted.

> So, you'd rather use something that is decades old because you can't/don't want to (ask to) install 1 new program on the server and take a weekend to rebuild your config file? Absolutely, yes. Because long story short, bash is TriedAndTrue® technology. Stuff that works. It takes a bit of dedication to be mastered at a decent level but it pays off immensely. It's so ubiquitous it is one of those tools that you can l…

"works" More like "better the devil you know" (if you can even say that much). Bash doesn't scale. Every shop larger than 1 has been burned by bash gotchas. Use a real scripting language and shell out to the commands and builtins when necessary. At my shop bash is strictly disallowed in production environments and we're all better off for it.

> Bash doesn't scale.

Thanks mate, I had a good laugh.

I wouldn't expect bash to scale anyway. That's not what it's meant for. It's meant for system administration task automation.

On a more serious note ...

In many occasions, the performance you get depends on how you tackle the problem you have, though. Even using bash and the tools from the unix toolbox, sometimes you can gain significant improvements on how you manage your data.

Anecdotal: I cannot remember the details, but I remember that rearranging the order of sorting and searching and removing duplicates (sort, sort -u, grep, uniq mainly) I saw a significant speedup.

Anecdotal (2): I cut the execution time of a night-running job from hours to minutes (tens of minutes, to be honest - but still less than an hour) just by slicing the size of a problem into smaller parts and by handling each slice in parallel (the machine had 48 cpus, but the problem was being solved "sequentially" on one cpu alone). I wrote some 30-50 lines of python, just to implement parallelism control: the rest of the problem was still handled with bash script. Partial results were reassembled at the end. Bash has coprocesses, so I might have handled that in bash as well, but python was more handy at the time (meh, i just wanted to optimize that problem).

What I am trying to say is that sometimes the "scaling" you get is justified by the size of the problem, sometimes it's not.

Re: Safe ways to do things in bash

#220

Post by znpy »

znpy wrote: post #219
opinionator1 wrote: post #200

Earlier quoted context omitted.

"works" More like "better the devil you know" (if you can even say that much). Bash doesn't scale. Every shop larger than 1 has been burned by bash gotchas. Use a real scripting language and shell out to the commands and builtins when necessary. At my shop bash is strictly disallowed in production environments and we're all better off for it.

> Bash doesn't scale. Thanks mate, I had a good laugh. I wouldn't expect bash to scale anyway. That's not what it's meant for. It's meant for system administration task automation. On a more serious note ... In many occasions, the performance you get depends on how you tackle the problem you have, though. Even using bash and the tools from the unix toolbox, sometimes you can gain significant improvements on how you m…

It's worth noting that bash is really a glue language to call other programs. If you mainly use the tools from the unix toolbox (i'm thinking of grep, for example) you really get the "scaling" (the performance) of native executable code.

Again, it really depends on how you handle your data.

Having a number of filters chained via pipes is really efficient, for example, when compared with looping over an array and executing some python/perl/ruby one-liners every time.

Post reply on HN