Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

281–290 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#281

Post by eschneider »

bitofhope wrote: post #197
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

There are workloads where shell scripts are the so-called right tool for a job . All too often I see people writing scripts in "proper" languages and calling os.system() on every other line. Shell scripts are good for gluing programs together. It's fine to use them for that.

1000% THIS. The trick, of course, is knowing when it's time to abandon shell for something more powerful, but that usually comes with experience.

Re: Shell script best practices, from a decade of scripting things

#282

Post by iso1631 »

IshKebab wrote: post #22
midasuni wrote: post #8

Earlier quoted context omitted.

If you need to follow these rules your script probably shouldn’t be written as a shell script.

Ha yeah someone should make a single lint "Your script is over 100 lines. You should rewrite it in a sane language!"

Start every script with the boilerplate

  #!/bin/bash
  if [[ `wc -l $0|cut -f 1 -d ' '` -gt 100 ]]
  then
  echo "No, this is too long!"
  exit
  fi

Re: Shell script best practices, from a decade of scripting things

#283

Post by iso1631 »

oars wrote: post #95

Do you guys think that Shell scripting will still be around in 20 years?

I have bash and perl scripts that keep major business critical services running that are about that age. Why would I think scripts I write today won't still be running in 20 years time?

Re: Shell script best practices, from a decade of scripting things

#284

Post by iso1631 »

casey2 wrote: post #128

More opinions 1. Bash shouldn't be used, not because of portability, but because its features aren't worth their weight and can be picked up by another command, I recommend (dash) any POSIX complaint shell (bash --posix included) so you aren't tempted to use features of bash and zsh that are pointless, tricky or are there for interactivity. Current POSIX does quite well for what you would use shell for. 2. Never use…

Just looked at two servers I'm sshed to - one redhat, one ubuntu. Neither has bash in /usr/bin/bash.

Re: Shell script best practices, from a decade of scripting things

#285

Post by blueflow »

coliveira wrote: post #187
hiepph wrote: post #165

Earlier quoted context omitted.

I can't really stand Bash's arcane syntax, it drains my brain power (and time of consulting manual) every time I have to work with it. Switching to Fish has been a breath of fresh air for me. I think some people who want to use only Bash need to open their conservative mind. All of my personal shell scripts now are converted to Fish. If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Of c…

This battle was lost a long time ago. Bash is the standard on most UNIX systems. If you change this reality, one might even start to try to think about writing in fish or some other new shell. But I will not even consider another shell for scripts that need to be run by other people.

POSIX shell is the standard, not bash.

Re: Shell script best practices, from a decade of scripting things

#286

Post by kdmccormick »

Cerium wrote: post #248
throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

"Use [[ ]] for conditions" Oh how I hate the double square bracket. It is the source of many head scratching bugs and time wasted. "The script works in my machine!" It doesn't work in production where we only have sh. It won't exit due to an error, the if statement will gobble the error. You only find the bug after enough bug reports hit that particular condition. After a couple shots to the foot I avoid double squar…

If I may ask, why do you only have sh in production?

Re: Shell script best practices, from a decade of scripting things

#287

Post by lbhdc »

usrbinbash wrote: post #264
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> What is the Shell paradigm? I would argue that it's line-oriented pipelines. Which python can do realitively well, by using the `subprocess` module. Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr` import subprocess as sp cat = sp.Popen( ["cat", "README.md"], stdout=sp.PIPE, ) grep = sp.Popen( ["grep", "#"], s…

> But on the other side of that coin its alot easier in python to do a complex regular expression

I am not sure I would agree. Sed fills this role quite nicely.

cat README.md | grep # | tr '[:lower:] [:upper:]' | sed 's/something/something_else/'

Re: Shell script best practices, from a decade of scripting things

#288

Post by maximus-decimus »

ducktective wrote: post #190
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

>However, if you instead organize all your data in a format that's sympathetic to line-oriented processing on stdin-stdout, then shell will work with you instead of against. Not even that is necessary. Just use structured data formats like json. If you are consuming some API that is not json but still structured, use `rq` to convert it to json. Then use `jq` to slice and dice through the data. dmenu + fzf + jq + curl…

How do you use dmenu for your shell script? to launch it? to prompt the user for input while it's running?

Do you have an example of a script you wrote?

Re: Shell script best practices, from a decade of scripting things

#289

Post by Spivak »

kotlin2 wrote: post #270
Spivak wrote: post #267

Earlier quoted context omitted.

Okay fair, I deserve that. I assumed it was obvious I meant arbitrary depth. Also d["a"] and d["a"]["b"] aren't 42.

If d["a"]["b"] is 42, then how could d["a"]["b"]["c"] also be 42? What you want doesn't make sense semantically. Normally, we'd expect these two statements to be equivalent d["a"]["b"]["c"] == (d["a"]["b"])["c"]

I mean you got it but it's something a lot of people want. The semantic reason for it is so you can look up an arbitrary path on a dict and if it's not present get a default, usually None. It can be done by catching KeyError but it has to happen on the caller side which is annoying. I can't make a real nested mapping that returns none if the keys aren't there.

    d = magicdict()
    is42 = d["foo"]["bar"]["baz"]
      # -> You can read any path and get a default if it doesn't exist.

    d["hello"]["world"] = 420 
      # -> You can set any path and d will then contain { "hello": { "world": 420 }
People use things like jmespath to do this but the fundamental issue is that __getitem__ isn't None safe when you want nested dicts. It's a godsend when dealing with JSON.

I feel like we're maybe too in the weeds, I should have just said "now have two expressions in your lambda."

Re: Shell script best practices, from a decade of scripting things

#290

Post by therealjumbo »

Cerium wrote: post #248
throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

"Use [[ ]] for conditions" Oh how I hate the double square bracket. It is the source of many head scratching bugs and time wasted. "The script works in my machine!" It doesn't work in production where we only have sh. It won't exit due to an error, the if statement will gobble the error. You only find the bug after enough bug reports hit that particular condition. After a couple shots to the foot I avoid double squar…

This should be fixed with a shebang and shellcheck. If your shebang is #!/bin/sh, shellcheck will complain loudly about bash-isms. If production is sh and doesn't have bash, there's quite a few other bash-ism you want to check for. You can run shellcheck in CI to check your scripts and return non-zero if they aren't clean, and you can force off warnings for lines that are ok.

EDIT: I should have said, "could be fixed once and for all", "should" is just my opinion.

Post reply on HN