Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell script best practices, from a decade of scripting things

sharats.me

Post reply on HN
Print view View original

261–270 of 500 posts

Re: Shell script best practices, from a decade of scripting things

#261

Post by replwoacause »

papascrubs wrote: post #233
Beltalowda wrote: post #66

Earlier quoted context omitted.

zsh is not a "cleaned-up bash"; it's more of a clone of ksh (closed source at the time), with some csh features added in, as well as their own inventions. bash and zsh appeared at roughly the same time, many features were added in zsh first and added to bash later (sometimes much later, and often never). This is kind of a good example of what I meant when people conflate "bash" with "shell". As for your larger point:…

The verbosity of PowerShell is overstated I think. You easily make POSH look as gnarly and esoteric as Bash if you so desire. That said, the majority of heavy lifting in POSH is done via methods these days (vs cmdlets). Your initial API query to snag the JSON might be via a cmdlet, but after that, you're slicing and dicing with real data structures. You can interact with them without having worry about whitespace or…

I agree with all of this. Well said.

Re: Shell script best practices, from a decade of scripting things

#262

Post by majkinetor »

l3uwin wrote: post #133
majkinetor wrote: post #114

Earlier quoted context omitted.

They don't feel primitive, they are primitive.

Yep, just as a screwdriver is. For certain jobs, that's all you need.

Indeed, except then you have to maintain screwdriver.

Re: Shell script best practices, from a decade of scripting things

#263

Post by Beltalowda »

ducktective wrote: post #201
Beltalowda wrote: post #23

Earlier quoted context omitted.

I've rewritten a lot of shell scripts with awk. Obviously it's not a good fit for everything, but when it is a good fit I found it a very pleasant experience. In spite of using Unix systems for 20 years I only learned awk a few years ago and I really beat myself up for not learning it earlier.

Convince me to up my game in awk! I only use it to select the n'th word in a csv-like line. Anything more than that, I need to search stackoverflow for the invocation. Don't you find its syntax cumbersome?

> Don't you find its syntax cumbersome?

Not really; just seems the same as most other dynamic languages. Awk does a lot of stuff for you (the "implied loop" your program runs in, field splitting) that's certainly possible (even easy) to replicate in Python or Ruby, but Awk it's just so much more convenient.

I use it for things like processing the Unicode data files, making some program output a bit nicer (e.g. go test -bench), ad-hoc spreadsheets, few other things. I got started with it as I needed to process some C header files and the existing script for that was in Awk; it worked pretty well for that too.

The Awk Programming Language book is pretty good. GNU Awk has a bunch of very useful extensions, but pretty much everything in the book still works and is useful today. You can get it at e.g. https://archive.org/details/awkprogrammingla00ahoa or https://github.com/teamwipro/learn_programing/blob/master/sh...

The GNU Awk docs are also pretty decent.

Re: Shell script best practices, from a decade of scripting things

#264

Post by usrbinbash »

xelxebar wrote: post #182

Hands down, shell scripting is one of my all time favorite languages. It gets tons of hate, e.g. "If you have to write more than 10 lines, then use a real language," but I feel like those assertions are more socially-founded opinions than technically-backed arguments. My basic thesis is that Shell as a programming language---with it's dynamic scope, focus on line-oriented text, and pipelines---is simply a different p…

> What is the Shell paradigm? I would argue that it's line-oriented pipelines.

Which python can do realitively well, by using the `subprocess` module.

Here is an example including a https://porkmail.org/era/unix/award (useless use of cat) finding all title lines in README.md and uppercasing them with `tr`

    import subprocess as sp
    cat = sp.Popen(
        ["cat", "README.md"],
        stdout=sp.PIPE,
    )
    grep = sp.Popen(
        ["grep", "#"],
        stdin=cat.stdout,
        stdout=sp.PIPE,
    )
    tr = sp.Popen(
        ["tr", "[:lower:]", "[:upper:]"],
        stdin=grep.stdout,
        stderr=sp.PIPE,
        stdout=sp.PIPE,
    )
    out, err = tr.communicate()
    print(out.decode("utf-8"), err.decode("utf-8"))
Is this more complicated than doing it in bash? Certainly. But on the other side of that coin its alot easier in python to do a complex regular expression (maybe depending on a command line argument) on one of those, using the result in an HTTP request via the `requests` module, packing the results into a digram rendered in PNG and sending it via email.

Yes, that is a convoluted example, but it illustrates the point I am trying to make. Everything outlined could probably done in a bash script, but I am pretty certain it would be much harder, and much more difficult to maintain, than doing this in python.

Bash is absolutely fine up to a point. And with enough effort, bash can do extremely complex things. But as soon as things get more complex than standard unix tools, I rather give up on the comfort of having specialiced syntax for pipes and filehandles, and write a few more lines handling those, if that means that I can do the more complex stuff easily using the rich module ecosystem of Python.

Re: Shell script best practices, from a decade of scripting things

#265

Post by kotlin2 »

Spivak wrote: post #256
kotlin2 wrote: post #250

Earlier quoted context omitted.

Now do an associative array containing another associative array.

Sometimes a you just have to accept a language's limitations. Try in Python to make a nested defaultdict you can access like the following. d = d["a"]["b"]["c"] # --> 42 Can't be done because it's impossible for user code to detect what the last __getitem__ call is and return the default. Edit: Dang it, I mean arbitrary depth.

    c = defaultdict(lambda: 42)
    b = defaultdict(lambda: c)
    a = defaultdict(lambda: b)
    a["a"]["b"]["c"]  # --> 42

Re: Shell script best practices, from a decade of scripting things

#266

Post by sumtechguy »

sharken wrote: post #156
orwin wrote: post #125

Earlier quoted context omitted.

I feel like powershell hides too much to be used regularly. I have a dozen of small shellscripts and aliases to do basically what PS help me to do when i work on windows (and some), but at least i know how it work behind. I had to work with Sencha/ExtJS early 2010. It was the same feeling. Yes, it is powerfull, but too much magic happen for something without a clear orientation (at the time, now it is used for data l…

I recently replaced a bit of code to look up locked files for a file share with SMB cmdlets to do the same. The performance difference was night and day. The biggest issue with PowerShell is that PowerShell Core is not yet default on Windows 10/11 and Windows Server. That should be Microsofts highest priority for PowerShell.

Then not hobble who can run powershell scripts out of the box. Which makes it seem like a dangerous tool. Then no one wants to use it. Some form of powershell has been there since win7. Yet in one of the versions they decided 'oh only admins can use this unless you run this special command'. So it makes me have to revert to using CMD scripts for some simple things. Because I do not want to have to walk whoever it is thru enabling powershell.

Re: Shell script best practices, from a decade of scripting things

#267

Post by Spivak »

kotlin2 wrote: post #265
Spivak wrote: post #256

Earlier quoted context omitted.

Sometimes a you just have to accept a language's limitations. Try in Python to make a nested defaultdict you can access like the following. d = d["a"]["b"]["c"] # --> 42 Can't be done because it's impossible for user code to detect what the last __getitem__ call is and return the default. Edit: Dang it, I mean arbitrary depth.

c = defaultdict(lambda: 42) b = defaultdict(lambda: c) a = defaultdict(lambda: b) a["a"]["b"]["c"] # --> 42

Okay fair, I deserve that. I assumed it was obvious I meant arbitrary depth.

Also d["a"] and d["a"]["b"] aren't 42.

Re: Shell script best practices, from a decade of scripting things

#268

Post by ghostoftiber »

Instead of implementing a -h or --help, consider using some code like "if nothing else matches, display the help". The asterisk is for this purpose.

  while getopts :hvr:e: opt
  do
      case $opt in
          v)
              verbose=true
              ;;
          e)
              option_e="$OPTARG"
              ;;
          r)
              option_r="$option_r $OPTARG"
              ;;
          h)
              usage
              exit 1
              ;;
          \*)
              echo "Invalid option: -$OPTARG" >&2
              usage # call some echos to display docs or something...
              exit 2
              ;;
      esac
  done

Re: Shell script best practices, from a decade of scripting things

#269

Post by [deleted] »

coliveira wrote: post #187
hiepph wrote: post #165

Earlier quoted context omitted.

I can't really stand Bash's arcane syntax, it drains my brain power (and time of consulting manual) every time I have to work with it. Switching to Fish has been a breath of fresh air for me. I think some people who want to use only Bash need to open their conservative mind. All of my personal shell scripts now are converted to Fish. If I want to run some POSIX-compatible script then I just use `bash scripts.sh` Of c…

This battle was lost a long time ago. Bash is the standard on most UNIX systems. If you change this reality, one might even start to try to think about writing in fish or some other new shell. But I will not even consider another shell for scripts that need to be run by other people.

[deleted]

Re: Shell script best practices, from a decade of scripting things

#270

Post by kotlin2 »

Spivak wrote: post #267
kotlin2 wrote: post #265

Earlier quoted context omitted.

c = defaultdict(lambda: 42) b = defaultdict(lambda: c) a = defaultdict(lambda: b) a["a"]["b"]["c"] # --> 42

Okay fair, I deserve that. I assumed it was obvious I meant arbitrary depth. Also d["a"] and d["a"]["b"] aren't 42.

If d["a"]["b"] is 42, then how could d["a"]["b"]["c"] also be 42? What you want doesn't make sense semantically. Normally, we'd expect these two statements to be equivalent

d["a"]["b"]["c"] == (d["a"]["b"])["c"]

Post reply on HN