Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

381–390 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#381

Post by raydiatian »

Shell scripting feels like scripting in cursive. It’s obfuscatory (sed? curl? grep? ssssuper descriptive) for the sake of thinking about solving problems like the elder generation. To make matters worse, you’re not even doing hard computer science things most of the time. You’re tweaking bits in files, uploading/downloading, searching, etc. It’s like having a butler who only understands your grocery list if it’s written in cursive.

I agree we need a shell scripting language, I disagree that bash zsh or anything that frequently uses double square brackets and awful program names is the epitome of shell scripting language design.

Re: Shell script best practices, from a decade of scripting things

#383

Post by mmh0000 »

ketanmaheshwari wrote: post #234
j0hnyl wrote: post #206

Earlier quoted context omitted.

I noticed that I became so much more quick after taking 1 hour to properly learn awk. Yes, it literally takes about 1 hour.

Awk is awesome but saying it literally takes 1 hour to properly learn it is a bit overselling.

What? The awk manual is only 827 highly technical pages[1]. If you can't read and interalize that in an hour, I suspect you're a much worse programmer than the OP.

[1] https://www.gnu.org/software/gawk/manual/gawk.html

For the sarcasm impared among us, everything above this, but possibly including this sentence is sarcasm.

Re: Shell script best practices, from a decade of scripting things

#384

Post by jb3689 »

snidane wrote: post #204
xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

Shell and SQL make you 10x productive over any alternative. Nothing even comes close. I've seen people scrambling for 1 hours to write some data munging, then spend another 1 hour to run it through a thread pool to utilize those cores , while somebody comfortable is shell writes a parallelized one liner, rips through GBs of data, and delivers the answer in 15 minutes. What Python is to Java, Shell is to Python. It sp…

I feel like the reasons for this are:

* Shell scripts force you to think in a more scalable way (data streams)

* Shell scripts compose rich programs rather than simplistic functions

* Shells encourage you to program with a rich, extensible feature set (ad-hoc I/O redirection, files)

The only times I don’t like shell scripts are when dealing with regex and dealing with parallelism

Re: Shell script best practices, from a decade of scripting things

#385

Post by memco »

jpitz wrote: post #238
throwaway2037 wrote: post #229

I'm surprised that no one mentioned a pair of tiny functions to log each command before it is run. Nicer versions also print a timestamp. Of course, this setup assumes: set -e echo_command() { echo echo '$' "$@" } echo_and_run_command() { echo_cmd "$@" "$@" } Then something like: main() { # For simple commands that do not use | etc. echo_and_run_command cp --verbose ... # More complex commands echo_command grep ... '…

It does not address timestamps, but set -x does this seamlessly without cluttering up your script. You can even run your script with sh -x script If you didn't always want the logging output.

In addition to set -x I have taken to wrapping my main entry point in some scripts where record keeping is helpful in a sub shell and pipe all output to a function that tees it's output to a log file after cleaning up escape sequences so I can generate a log file without having to annotate every line with some kind of wrapper:

  #!/usr/bin/env bash
  (
    foo
    bar
  ) 2>&1 | print_and_log "$logfile"

Re: Shell script best practices, from a decade of scripting things

#386

Post by xwowsersx »

Can someone enlighten me on the

  cd "$(dirname "$0")"
part of this? This is changing to the directory of where the script is all cases?

EDIT: I should've just tested this to see :) I did and it does exactly that. Very helpful. I didn't realize $0 is always the first argument. Kind of like how `self` is the first implicit argument in OOP methods?

Re: Shell script best practices, from a decade of scripting things

#387

Post by boredtofears »

tejtm wrote: post #379
boredtofears wrote: post #324

Earlier quoted context omitted.

yeah thats exactly right. it may only take an hour to learn, but every time i need to use awk it seems like i have to spend an hour to re-learn its goofy syntax.

alas this is true, I never correctly recall the order of particular function args as they are fairly random, still beats the alternative of having to continually internalize entire fragile ecosystems to achieve the same goal.

yeah you're definitely right. im sure if it was something i had to use more consistently i'd be able to commit it to memory. maybe...

Re: Shell script best practices, from a decade of scripting things

#388

Post by GuB-42 »

majkinetor wrote: post #380
GuB-42 wrote: post #151

Earlier quoted context omitted.

I tried PowerShell, hated it. The idea of manipulating objects instead of text streams is interesting, and avoids most of the footguns sh/bash have, but you also lose in flexibility. One reason is that there are thousands of command line tools in the UNIX ecosystem that process text streams and are designed to work with shells like bash. You have much less options when you are processing PowerShell objects. Note: I t…

ls -File | % { $fh = Get-FileHash $_ SHA1; if ($_.Name -match $fh.hash) {$_} }

Doesn't work, it is not recursive and it was a CRC, not SHA1 but I probably could have found a solution starting from there.

Anyways, it was a long time ago, maybe I will give PowerShell a retry at some point.

Re: Shell script best practices, from a decade of scripting things

#389

Post by pdntspa »

rcthompson wrote: post #326
hnlmorg wrote: post #227

Earlier quoted context omitted.

> getting the shell quoting hell right Shameless plug coming, it this has been a pain point for me too. I found the issue with quotes (in most languages, but particularly in Bash et al) is that the same character is used to close the quote as is used to open it.m. So in my own shell I added support to use parentheses as quotes in addition to the single and double quotation ASCII symbols. This then allows you to nest…

One of my favorite Perl features that has been disappointingly under-appropriated by other languages is quoting with q(...).

This is one of my favorite features of Ruby!

Though Ruby makes it confusing AF because there are two quoting types for both strings and symbols, and they're different. (%Q %q %W %w %i) I can never remember which does which.... the letter choice feels really arbitrary.

Re: Shell script best practices, from a decade of scripting things

#390

Post by majkinetor »

GuB-42 wrote: post #388
majkinetor wrote: post #380

Earlier quoted context omitted.

ls -File | % { $fh = Get-FileHash $_ SHA1; if ($_.Name -match $fh.hash) {$_} }

Doesn't work, it is not recursive and it was a CRC, not SHA1 but I probably could have found a solution starting from there. Anyways, it was a long time ago, maybe I will give PowerShell a retry at some point.

Ofc. it works. If on linux, replace ls with `gci`. Recursion is done with ls -Recurse. Install-Package CRC. Its still single liner.
Post reply on HN