Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

301–310 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#301

Post by 0xbadcafebee »

Spivak wrote: post #252
throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

> Only use that for bashisms where there's no POSIX alternative This seems like really bad advice because the number of people writing bash massively massively outnumbers the people writing sh. Regex matching, glob matching, proper parsing, &&/||, no need to quote. I would say the opposite, enjoy all the bashisms like (( )) [[ ]], numeric for loops, extended globs, brace expansion, ranges, OH GOD YES VARIABLE REFEREN…

Almost nobody knows all those bashisms. Nobody on the team will be able to understand it, edit it. Avoid being fancy if there's an uglier, simpler thing.

Re: Shell script best practices, from a decade of scripting things

#302

Post by cryptonector »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

I've written a lot of shell scripts. I have my own best practices that work for me. I don't like it one bit. I mean, it's enjoyable to write shell scripts, it's just not enjoyable to deal with them long-term.

Re: Shell script best practices, from a decade of scripting things

#303

Post by graton »

Author mentions using xtrace aka `set -x`. If using xtrace I highly recommend doing:

    export PS4='+ ${BASH_SOURCE:-}:${FUNCNAME[0]:-}:L${LINENO:-}:   '
This will then append the filename, function name, and line number to the command being executed. Can make it much easier to find where exactly something is happening when working with larger bash scripts.

Re: Shell script best practices, from a decade of scripting things

#304

Post by IYasha »

rethab wrote: post #100
IYasha wrote: post #84

Mostly agree, but I add more. 1. end all your lines C-style; this may save your life many times; 2. declare -is variables and -r CONSTANTS at the beginning, again, C-style; 3. print TIMESTAMP="$(date +%Y-%m-%d\ %H:%M:%S)"; where appropriate if your script logs its job; 4. Contrary to OPs reommendation I strongly try to stick to pure SH compatibility in smaller acripts so they can run on routers, TVs, androids and oth…

how do you make sure your scripts are SH compatible?

Make sure? Well, aside from keeping sh feature subset in my head, I usually run them like "sh myscript.sh" on target or limited environment (they have #!/bin/sh shebang) for testing. Other people here probable have better suggestions though. )

Re: Shell script best practices, from a decade of scripting things

#305

Post by snidane »

foobarian wrote: post #276
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.

That's a hardware management question. The optimized binary used in my shell script still runs orders of magnitude faster and cheaper if you orchestrate 100 machines for it than any Hadoop, Spark, Beam, Snowflake, Redshift, Bigquery or what have you.

That's not to say I'd do everything in shell. Most stuff fits well into SQl, but when it comes to optimizing processing over TB or PB scale, you won't beat shell+massive hw orchestration.

Re: Shell script best practices, from a decade of scripting things

#306

Post by 0xbadcafebee »

asicsp wrote: post #37
selectnull wrote: post #34

> Use the .sh (or .bash) extension for your file. It may be fancy to not have an extension for your script, but unless your case explicitly depends on it, you’re probably just trying to do clever stuff. Clever stuff are hard to understand. I don't agree with this one. When I name my script without extension (btw, .sh is fine, .bash is ugly) I want my script to look just like any other command: as a user I do not care…

My thumb rule is no extension if the script goes to the local bin folder and `.sh` otherwise. Beyond syntax highlighting, the extension also helps for wildcard matching for file operations (`ls`, `cp`, `for` loop, etc).

shellcheck *.sh FTW

Re: Shell script best practices, from a decade of scripting things

#307

Post by generalizations »

ndsipa_pomu wrote: post #120
suprjami wrote: post #55

Use a linter. Pass all scripts through https://www.shellcheck.net/ or use `shellcheck` on the commandline. Learn the things it tells you and implement them in future scripts.

That should be the zeroth rule of all shell/bash scripting. I'm almost tempted to put in a self-linting line in scripts so that they won't run unless shellcheck passes completely. (It would be unnecessary to lint the same script every time it's called though, so it's not a serious suggestion). There should be an option in bash to auto-lint scripts the first time that they're called, but I don't know how the OS should…

It would be simpler to modify shellcheck to add flags to shellcheck that limit the kinds of warnings it produces, and then just run it on every invocation of the script. That keeps everything local and deterministic.

Re: Shell script best practices, from a decade of scripting things

#308

Post by kotlin2 »

Spivak wrote: post #289
kotlin2 wrote: post #270

Earlier quoted context omitted.

If d["a"]["b"] is 42, then how could d["a"]["b"]["c"] also be 42? What you want doesn't make sense semantically. Normally, we'd expect these two statements to be equivalent d["a"]["b"]["c"] == (d["a"]["b"])["c"]

I mean you got it but it's something a lot of people want. The semantic reason for it is so you can look up an arbitrary path on a dict and if it's not present get a default, usually None. It can be done by catching KeyError but it has to happen on the caller side which is annoying. I can't make a real nested mapping that returns none if the keys aren't there. d = magicdict() is42 = d["foo"]["bar"]["baz"] # -> You ca…

What languages allow such a construct? It seems like it would be super confusing if these two code samples produced different values:

    # One
    a = d["a"]["b"]["c"]
    
    # Two
    a = d["a"]["b"]
    b = a["c"]

Re: Shell script best practices, from a decade of scripting things

#309

Post by blueflow »

chubot wrote: post #298
blueflow wrote: post #285

Earlier quoted context omitted.

POSIX shell is the standard, not bash.

That ship has sailed, because busybox ash and dash continually implement some of bash features and semantics, which come from ksh. And OSH implements almost all of bash That is, The posix shell spec is missing a lot of reality. It’s not very actively maintained, unfortunately The canonical example is not having local vars, which basically every shell supports

"Everybody using some extensions" is not a contradiction to "The standard is the baseline".

> It’s not very actively maintained, unfortunately

Because that's how standards works. If there is enough interest a new standard will be made, but right now, bash isn't it.

Re: Shell script best practices, from a decade of scripting things

#310

Post by snidane »

kotlin2 wrote: post #250
Spivak wrote: post #246

Earlier quoted context omitted.

I mean they're not that bad. declare -A mydict( [lookma]=initalization ) mydict[foo]=bar echo "${mydict[foo]}" list=() list+=(foo bar baz) echo "${list[0]}"

Now do an associative array containing another associative array.

Flatten the damn thing and process it relationally. Linear data scans and copying are so fast on modern hardware that it doesn't matter. It's counterintuitive for people to learn that flattened nested structure with massive duplication still processes faster than that deeply nested beast because you have to chase pointers all over the place. Unfortunately that's what people learn at java schools and they get stuck with that pointer chasing paradigm for the rest of their careers.
Post reply on HN