Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safe ways to do things in bash

github.com

Post reply on HN
Print view View original

121–130 of 255 posts

Re: Safe ways to do things in bash

#121

Post by freedomben »

antris wrote: post #113
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

What is your opinion on using other languages to augment bash? Awk, perl etc.

I personally consider awk, sed, grep, cut, etc. to be "part" of bash. I don't think I ever write a script without invoking those venerable tools at least once. I often have many pipes, such as

  some_command \
    | grep -E 'some.*regex$' \
    | sed -e 's/erase_text//g' \
    | sed -e 's/erase_more//g' \
    | awk '{ print $2 }' \
    | cut -d ':' -f 1
I used to reach for Perl one-liners a lot, and still do sometimes when I need a gross regex that `sed -E` can't handle (See Perl Pie[1]), but Ruby has been making an appearance more and more often for handy one-liners. Roblog has a great post on using Ruby [2].

If the script is complex enough and I'm willing to take a dependency on Ruby, that is usually where I turn. Where that line is does change from time to time. Ruby makes it super trivial to call shell commands and process them easily as well, which makes it easy to integrate into existing scripting [3].

Hopefully that answered your question :-)

[1] http://technosophos.com/2009/05/21/perl-pie-if-you-only-lear...

[2] https://robm.me.uk/ruby/2013/11/20/ruby-enp.html

[3] https://stackoverflow.com/a/2400/2062384

Re: Safe ways to do things in bash

#122

Post by freedomben »

themodelplumber wrote: post #111
freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

> 6. You will not regret getting really good at shell script. You'll have to take my word for it now because you don't know what you're missing. Stories! C'mon, it was a long Monday :-)

Haha. Mostly it is dozens, probably hundreds of little things over time when turning to the shell proved highly productive and effective. The shell is surprisingly really, really good at processing text (using the good 'ol unix tools), and it's amazing how often "text processing" type problems come up. Whether you are doing find/replace on source code, looking for where certain strings are used/defined, or curling some page and extracting data out of it, the shell can often provide very quickly and effectively. The bonus feature is that people are often very impressed when you whip out the shell skills and nail something they thought would be difficult.

I also started capturing most tasks as bash functions or aliases in my `~/.bashrc` file. This doubles as both a handy reference for me to look at later to remember how to do something, and automating simple things.

Re: Safe ways to do things in bash

#123

Post by groovy2shoes »

JoshTriplett wrote: post #11
nisa wrote: post #6

Earlier quoted context omitted.

if you need arrays use mksh - https://www.mirbsd.org/mksh.htm - if you need complex scripting with small footprint and standard compliance avoid bash and use mksh.

You can count on bash being present on almost any Linux system, other than tiny embedded ones. You can't count on mksh being anywhere unless you install it. Doesn't matter if it's better or worse, it isn't omnipresent.

In addition to bash itself being widespread, most of the common "bashisms" are also supported by zsh and most variants of ksh (and many systems actually use some variant of ksh disguised as /bin/sh). This is unsurprising given that bash itself was originally meant to be a clone of ksh. The places where they differ tend to cluster around uncommon/newer features (like coprocesses), certain syntactic quirks (but if you're aware of them they're usually easy to work around), and interactive features (which are mostly irrelevant for scripts). If you're writing simple scripts (or you're writing complex ones in a careful way), you can reasonably "target" bash and expect them to work on other common shells.

If you're going to be distributing shell scripts, it's probably a good idea to test them on a handful of common shells anyway. If portability between shells isn't something you're worried about, it's exactly as reasonable to ask users to install bash as it is to ask them to install mksh or zsh (or even python or lua or whatever) anyway. If you're most comfortable with bash, just use bash!

But for the love of god and all that is holy, please don't write csh scripts ;)

Re: Safe ways to do things in bash

#124

Post by freedomben »

freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

Can't edit original, but to answer the question regarding good sources for learning, I highly recommend The Linux Command Line . Great book: http://linuxcommand.org/tlcl.php

Re: Safe ways to do things in bash

#125

Post by xixixao »

If your everyday language is JavaScript, use shelljs. It’s great. You’ll get portable scripts in no time. Sure it’s “slow”, in a way that likely doesn’t matter because of what you’re calling from the script.

Re: Safe ways to do things in bash

#126

Post by foxylad »

geofft wrote: post #118
saagarjha wrote: post #14

Earlier quoted context omitted.

> You are source controlling your shell scripts right? My personal heuristic for shell scripts are that if I care enough about them to put them under source control, they shouldn't be a shell script.

I have seen entirely too many Python scripts that are poor reimplementations of shell scripts that don't get the edge cases right, have more security vulnerabilities, etc. than a shell script a fifth of the size. There are things you simply can't do in other languages without excessive verbosity. (One that came up at my workplace recently is using Use the right tool for the job, and then put it in version control (an…

> without excessive verbosity

If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

Re: Safe ways to do things in bash

#127

Post by zwischenzug »

freedomben wrote: post #69

I've written a ridiculous amount of shell script in my day, especially when doing "devops" before we had a term like "devops" to describe it. I've fallen in almost every pit bash has. With that background, here is my opinion. 1. This article contains excellent advice and should be starred for later retrieval. 2. Having basic scripting skills will make you a way better programmer. Many times I've done huge refactors a…

I did a talk on this a while ago:

https://www.youtube.com/watch?v=pb3k0sGKrjQ&t=457s

'Take bash seriously'

Re: Safe ways to do things in bash

#128

Post by kiallmacinnes »

ddtaylor wrote: post #117
koolba wrote: post #95

Earlier quoted context omitted.

> It's funny how many things that seem kind of incredible for a pure bash solution, ... My go to example of “Bash can do whaaat?!” is the source for xip.io. A custom DNS server written in a handful of lines of Bash! https://github.com/basecamp/xip-pdns/blob/master/bin/xip-pdn...

I have made a lot of DNS products and doing the same thing happening here is a few lines of code in most any language. This code, while entertaining, cannot handle very simple DNS packets because of its "compression encoding.

This is ran behind PowerDNS, which handles the compression prior to handing off to this code, so that shouldn't be a problem.

Re: Safe ways to do things in bash

#129

Post by geofft »

foxylad wrote: post #126
geofft wrote: post #118

Earlier quoted context omitted.

I have seen entirely too many Python scripts that are poor reimplementations of shell scripts that don't get the edge cases right, have more security vulnerabilities, etc. than a shell script a fifth of the size. There are things you simply can't do in other languages without excessive verbosity. (One that came up at my workplace recently is using Use the right tool for the job, and then put it in version control (an…

> without excessive verbosity If no-one (including you, unless you dream in bash) is ever going to modify or extend the script, OK. Otherwise python's readability/maintainability trumps excessive verbosity every time.

By "excessive verbosity" I do mean poor readability and poor maintainability. If you're invoking 5 commands and most of your Python code is wiring up the commands to each other right, just write 5 lines of shell, don't make me pull up the subprocess docs to see if your 30 lines of Python are doing the same thing and how to make a small change without risking pipes deadlocking.

Python is readable and maintainable when it's doing the things Python is suited towards doing. Running lots of external processes is not one of those. This is not a complaint or an insult to Python as a language (which I use regularly!), it is just a statement that different languages have different strengths and you should use the right tool for the job.

If you know of a good Python library that handles things like shell pipelines, <(...), and automatic creation of process groups (so that signal handling does the right thing), I'd be extremely interested, because I would like to use Python for these use cases. But it's currently the wrong thing for maintainable code for this one use case, and there is a very good language that handles this use case very well and is extremely stable and widely deployed.

Re: Safe ways to do things in bash

#130

Post by dredmorbius »

rkeene2 wrote: post #112
dredmorbius wrote: post #56

Earlier quoted context omitted.

Termux rewrites scripts via termux-fix-shebang.

It now also does rewriting on the fly for certain shebang lines, as part of its C library, so unmodified scripts can run.

I didn't know that, thanks.
Post reply on HN