Common shell script mistakes (2008) • Hacker News Board

Re: Common shell script mistakes (2008)

#41

Post by naivepiano » Sat, Jun 04, 2016, 11:56 AM UTC

s_kilk wrote: post #36

stewartbutler wrote: post #30

So many edge cases. I prefer to just use perl if I am doing anything more complicated than making a wrapper for some other executable. Harder to screw up, easier to test, same expressiveness, better portability (works with all shells and with windows), plus you get regular expressions for free. Python is a good alternative as well.

Perl is brutally under-rated as a scripting tool these days. I suppose it's passed into software orthodoxy by this point, the notion that perl is a uniformly horrific tool akin to nuclear waste.

I think that this is because people are finding Python equally well suited for such tasks and more modern. Perl (easy to) obscure syntax is not helping either.

Re: Common shell script mistakes (2008)

#42

Post by cm3 » Sat, Jun 04, 2016, 12:18 PM UTC

brbsix wrote: post #34

pwd_mkdb wrote: post #21

Earlier quoted context omitted.

my personal belief is that anything one can do in bash i can do in sh. not sure if that's really true in practice, but that's my belief. i never use bashisms because i do not know what they are or how to use them.

You're just having a laugh at the OP, right? In case you're not, here are some "Bashisms" that really suck to be without: * built-in regex support (e.g. `[[ $var =~ ^1\.2\.[34]$ ]]`) * process substitution (e.g. `diff * indexed and associative arrays Some of this can be worked around by shelling off to grep for regular expression matching or awk for arrays, but Bash makes things so much cleaner and maintainable.

But once you adopt bash-only features, you're losing the main argument for a shell script: portable scripting without the need to first install something to get something else running. Once you require Bash, it's equally easy to demand Perl and that will provide a much richer scripting experience.

Re: Common shell script mistakes (2008)

#43

Post by nailer » Sat, Jun 04, 2016, 12:20 PM UTC

pmarin wrote: post #33

pvg wrote: post #8

The google shell style guide linked now lives here and is probably an even better starting point than this article: https://google.github.io/styleguide/shell.xml

The google shell style uses bash which is not portable, one of the main point of the OP article.

Bash is the default shell on the most popular server nix (Linux) and the most desktop nix (OS X, though it's always a slightly old version).

If you use OpenBSD and hate ports, then maybe ignore all the good things bash brings and write everything in Bourne shell. If not, then write bash. Just fire it up with '/usr/bin/env bash' to label it correctly.

Re: Common shell script mistakes (2008)

#44

Post by mbrock » Sat, Jun 04, 2016, 12:29 PM UTC

daveguy wrote: post #17

gizi wrote: post #11

Bash is the love of my life! I have been working for years on this problem now (not full-time of course), gradually moving in the direction of finally being able to challenge this: "Inappropriate use shell is the main domain specific language designed to manipulate the UNIX abstractions for data and logic, i.e. files and processes. ... Correspondingly, please be wary of writing scripts that deviate from these abstrac…

> My personal belief is that everything that you can do in other scripting languages, you can also do in Bash, only better. Matrix multiplication?

Effective bash programming involves invoking other Unix commands. That's what the shell is for, after all. I don't know off the top of my head what you'd use for matrix multiplication; I guess it depends.

For another example, jq and jshon are excellent tools for querying, transforming, or creating JSON values.

My experience is that programming by composing Unix programs that communicate via pipes and files can be very pleasant and productive.

Sometimes I lament aspects of bash syntax and semantics, but it's also much more convenient than other scripting languages for tasks related to spawning programs and pipelines and working with files.

One of my recent projects that I've been having a lot of fun with is a kind of continuous integration system built mostly with bash scripts. It involves a lot of calls to git and Docker, process manipulation, file descriptor redirection, and file system manipulation—and bash makes all this pretty easy and concise.

Re: Common shell script mistakes (2008)

#45

Post by logicallee » Sat, Jun 04, 2016, 12:43 PM UTC

s_kilk wrote: post #36

stewartbutler wrote: post #30

So many edge cases. I prefer to just use perl if I am doing anything more complicated than making a wrapper for some other executable. Harder to screw up, easier to test, same expressiveness, better portability (works with all shells and with windows), plus you get regular expressions for free. Python is a good alternative as well.

Perl is brutally under-rated as a scripting tool these days. I suppose it's passed into software orthodoxy by this point, the notion that perl is a uniformly horrific tool akin to nuclear waste.

you can't learn perl 5 (as a new programmer) because of perl 6, and the last time I did a search for a perl 6 solution it didn't work for me, I actually ended up installing perl 5 and just doing it that way. I remember now, I literally googled "perl 6 installer windows" and clicked the first link and within 10 minutes couldn't install it. I think the installer just hangs - http://rakudo.org/how-to-get-rakudo/

so I just installed perl 5 and did it my usual way. But if I didn't know Perl already it would be a non-option. it's just not a language anymore, it's a legacy thing that if you already learned you can sometimes still use.

Re: Common shell script mistakes (2008)

#46

Post by bogomipz » Sat, Jun 04, 2016, 1:23 PM UTC

cm3 wrote: post #42

brbsix wrote: post #34

Earlier quoted context omitted.

You're just having a laugh at the OP, right? In case you're not, here are some "Bashisms" that really suck to be without: * built-in regex support (e.g. `[[ $var =~ ^1\.2\.[34]$ ]]`) * process substitution (e.g. `diff * indexed and associative arrays Some of this can be worked around by shelling off to grep for regular expression matching or awk for arrays, but Bash makes things so much cleaner and maintainable.

But once you adopt bash-only features, you're losing the main argument for a shell script: portable scripting without the need to first install something to get something else running. Once you require Bash, it's equally easy to demand Perl and that will provide a much richer scripting experience.

When it this last time you logged into a box and /bin/bash wasn't an option? Let me guess 1999 on a SPARC box running Sun Solaris?

Re: Common shell script mistakes (2008)

#47

Post by jerf » Sat, Jun 04, 2016, 1:35 PM UTC

naivepiano wrote: post #41

s_kilk wrote: post #36

Earlier quoted context omitted.

Perl is brutally under-rated as a scripting tool these days. I suppose it's passed into software orthodoxy by this point, the notion that perl is a uniformly horrific tool akin to nuclear waste.

I think that this is because people are finding Python equally well suited for such tasks and more modern. Perl (easy to) obscure syntax is not helping either.

If you're doing a really grungy, grotty text mangling task in a UNIX pipeline, suddenly all those crazy Perl features start making sense. The reason why Perl is arguably not the best choice for other tasks, such as creating large systems, is precisely that it's actually got many distinct advantages over Python for that sort of task. It's also really good for "I have a UNIX pipeline but this one chunk of it needs a database and this other chunk needs a JSON file and a few other things that shell just can't do".

For instance, for the first time in 15+ years, I recently had a reason to use Perl's half-deprecated, off-handedly-included form printing support, and, you know what? It's really nice if that's what you want. Many dozens of lines of code replaced by what is basically a picture of what I want.

Python is a great language, but every once in a while "explicit is better than implicit", which is generally a philosophy I agree with wholeheartedly, results in a lot more "explicit", in the form of "code I had to write", than you might be looking for. (Or, alternatively, you start writing Python that is a lot more implicit, which isn't exactly a huge victory for Python here.)

Nowadays I'd also suggest that if you find yourself having a lot of those tasks, you need to ask some questions about how you're storing your data. But pretty much everybody ends up with some of these somewhere; it's hard to eliminate them, and not really worth it (due to diminishing returns).

Re: Common shell script mistakes (2008)

#48

Post by jerf » Sat, Jun 04, 2016, 1:41 PM UTC

int0x80 wrote: post #26

majestik wrote: post #19

Earlier quoted context omitted.

Ok, I'll bite. >> My personal belief is that everything that you can do in other scripting languages, you can also do in Bash, only better. 1) Native JSON, XML 2) Classes, namespacing, objects 3) Multiprocessing, multithreading 4) Performance 5) Package management 6) Portability 7) Documentation 8) Runtime debugging (!set -x) I'm too tired to continue.

>3) Multiprocessing IMO shell makes it very easy to work with multiple process (&). It's built in and natural. >4) Performance If you are carefull and know what you're doing, you can achive very good performance with the shell. Usually, better performance is achived processing less data, ie being inteligent. Rarely depends on the language (unless you care about cycle level performance, then yes :). >6) Portability I…

Shell makes it easy to spawn multiple processes. It makes it reasonably easy to read those processes' standard out or standard error, though it's not that much fun to try to do both at the same time while keeping them distinct. [1]

It pretty much doesn't do anything else that you might want to do with multiple processes, though, and it tends to encourage multiple processes to communicate via text which is a problematic limitation that one often finds oneself "working around".

Shell is really powerful, but it hits a certain limit of what kind of tasks it can do and it hits that limit hard, and that's why when one imagines orchestrating many processes on a machine to do some task, to say nothing of orchestrating many processes on many machines, you don't see solutions based on shell, and indeed the very idea is laughable. Shell is best used by making sure it stays firmly restricted to the domain it shines in and not so much as trying to dip a toe into the spaces where it is not.

[1]: Note "not much fun" != "can't". Shell is fundamentally written around the idea that a process has one stream STDOUT that may go to other processes, and one stream STDERR which is generally intended to go to the console (or other user output like a log) no matter how complicated the pipeline. While you can get both streams and do things to them, you're starting to fight shell, which really wants to create pipelines with one "through" path with no branches out.

Re: Common shell script mistakes (2008)

#49

Post by [deleted] » Sat, Jun 04, 2016, 2:11 PM UTC

nailer wrote: post #43

pmarin wrote: post #33

Earlier quoted context omitted.

The google shell style uses bash which is not portable, one of the main point of the OP article.

Bash is the default shell on the most popular server nix (Linux) and the most desktop nix (OS X, though it's always a slightly old version). If you use OpenBSD and hate ports, then maybe ignore all the good things bash brings and write everything in Bourne shell. If not, then write bash. Just fire it up with '/usr/bin/env bash' to label it correctly.

[deleted]

Re: Common shell script mistakes (2008)

#50

Post by cm3 » Sat, Jun 04, 2016, 2:16 PM UTC

bogomipz wrote: post #46

cm3 wrote: post #42

Earlier quoted context omitted.

But once you adopt bash-only features, you're losing the main argument for a shell script: portable scripting without the need to first install something to get something else running. Once you require Bash, it's equally easy to demand Perl and that will provide a much richer scripting experience.

When it this last time you logged into a box and /bin/bash wasn't an option? Let me guess 1999 on a SPARC box running Sun Solaris?

If you limit yourself to either a popular Linux distro or one of two Unixes/Unix-likes, then bash can be available out of the box. Just as C is not C++, shell is not bash, so a shell script is what runs on (d)ash, busybox, toybox, ksh, bash, zsh, etc., without modifications. If it requires bash or zsh, then it's not a shell script but a bash/zsh/fish script. To name a popular non-Linux OS, take a fresh FreeBSD or OpenBSD install. No bash to be found, unless installed via ports and rightfully so. That said, I use bash myself all the time as an interactive shell but /bin/sh is not bash. sh (including bash and zsh) are terribly hard to write correct and resilient scripts in, and even rc is much saner to script in.