Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

271–280 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#271

Post by jjice »

ducktective wrote: post #201
Beltalowda wrote: post #23

Earlier quoted context omitted.

I've rewritten a lot of shell scripts with awk. Obviously it's not a good fit for everything, but when it is a good fit I found it a very pleasant experience. In spite of using Unix systems for 20 years I only learned awk a few years ago and I really beat myself up for not learning it earlier.

Convince me to up my game in awk! I only use it to select the n'th word in a csv-like line. Anything more than that, I need to search stackoverflow for the invocation. Don't you find its syntax cumbersome?

> I only use it to select the n'th word in a csv-like line.

If it's not a CSV with quotes to allow for commas inside values (which I think AWK will also fail in), you can use `cut`.

    cut -d, -f n
To delimit on a comma and select the n'th field. Reads a bit easier IMO for that common AWK use case (which I used to use AWK for all the time).

Re: Shell script best practices, from a decade of scripting things

#272

Post by ndsipa_pomu »

dotancohen wrote: post #194
marklgr wrote: post #184

Earlier quoted context omitted.

> "If you have to write more than 10 lines, then use a real language" I swear, there should be a HN rule against those. It pollutes every single Shell discussions, bringing nothing to them and making it hard for others do discuss the real topic.

There are three numbers in this industry: 0, 1 and infinity. Any other number - especially when stated as a rule, limitation, or law - is highly suspect.

0, 1, 3 and infinity

Re: Shell script best practices, from a decade of scripting things

#273

Post by devonkim »

j0hnyl wrote: post #206
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

It takes a lot less time to learn to be fairly productive with awk than with, say, vi / vim. Over time I've realized that gluing these text manipulation tools together is only an intermediate step toward learning how to write and use them in a manner that is maintainable across several generations of engineers as well as portable across many different environments, and that's still a mostly unsolved problem IMO for not just shell scripts but programming languages in general. For example, the same shell script that does something as seemingly simple as performing a sha256 checksum on macOS won't work on most Linux distributions. So in the end one winds up writing a lot of utilities all over again in yet another language for the sake of portability which ironically hurts maintainability and readability for sure because it's simply more code that can rot.

Re: Shell script best practices, from a decade of scripting things

#274

Post by wazoox »

selectnull wrote: post #34

> Use the .sh (or .bash) extension for your file. It may be fancy to not have an extension for your script, but unless your case explicitly depends on it, you’re probably just trying to do clever stuff. Clever stuff are hard to understand. I don't agree with this one. When I name my script without extension (btw, .sh is fine, .bash is ugly) I want my script to look just like any other command: as a user I do not care…

Also an extension will prevent execution from cron.d on Debian-based systems.

Re: Shell script best practices, from a decade of scripting things

#275

Post by foobarian »

ketanmaheshwari wrote: post #234
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

All you need to do is learn that cmd | awk '{ $5 }' will print out the 5th word as delimited by one or more whitespace characters. Regexes support this easily but are cumbersome to write on the command line.

Re: Shell script best practices, from a decade of scripting things

#276

Post by foobarian »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes.

This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.

Re: Shell script best practices, from a decade of scripting things

#277

Post by ekianjo »

foobarian wrote: post #276
snidane wrote: post #204

Earlier quoted context omitted.

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

> while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.

usually you use specific frameworks for that, not pure Python.

Re: Shell script best practices, from a decade of scripting things

#278

Post by foobarian »

ekianjo wrote: post #277
foobarian wrote: post #276

Earlier quoted context omitted.

> while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. This also works up to a point where those GBs turn into hundreds of GBs, or even PBs, and a proper distributed setup can return results in seconds.

usually you use specific frameworks for that, not pure Python.

I suppose the Python side is a strawman then - who would do that for a small dataset that fits on a machine? Or have I been using shell for too long :-)

Re: Shell script best practices, from a decade of scripting things

#279

Post by jlg23 »

throwawaaarrgh wrote: post #203

"Use set -o errexit" Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail. In particular, scripts you source into your shell should not use set options to change the shell's default behavior. "Prefer to use set -o nounset." ALWAYS use this option. You can test for a variable that might not be set with "${FOO:-}". Ther…

>> "Use set -o errexit"

> Only if it doesn't matter that the script fails non-gracefully. Some scripts are better to either have explicit error handling code, or simply never fail.

Then handle those errors explicitly. The above will catch those error that you did not think about.

Re: Shell script best practices, from a decade of scripting things

#280

Post by jayd16 »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Eh, this is true but I dont think its because of the programming model of bash. I feel like this is conflating the *nix ecosystem with bash. If every programming language was configured by default and had access to standard unix tools with idiomatic bindings, Shell's advantages would be greatly reduced. You still get a scripting language with some neat tricks but I don't think I would reach for it nearly as often if other things were an option.

And sure sure you can call any process from a language but the assumptions are different. No one wants to call a Java jar that has a dependency on the jq CLI app being available.

Post reply on HN