Live data from Hacker News

Techniques I use to create a great user experience for shell scripts

nochlin.com

251–260 of 281 posts

Re: Techniques I use to create a great user experience for shell scripts

#251

Earlier quoted context omitted.

> it seems odd that an experienced script writer would be falling foul of shellcheck being pedantic about what you're writing. It's not simply being pedantic, it is wrong . Your writing gives the impression that the tool is infallible. If I was new to writing shell scripts, shellcheck is clearly a wise choice. The language is loaded with footguns. But as someone who has been writing scripts for decades, I already kno…

I do agree about ShellCheck being wrong sometimes - that's why I mentioned the method of disabling it for specific lines with a comment. When I first started using ShellCheck, it was highlighting lots of footguns that I wasn't aware of, but nowadays, it's very rare for it to spit a warning out at me - usually just for something like it not following a script or stating that a variable wasn't defined when it was. I th…

> I think the huge number of footguns is what makes BASH scripting fun.

We have a completely different definition of fun.

I really only use bash when I need to chain a few commands. As soon as there is more complex logic I move to some programming language.

Re: Techniques I use to create a great user experience for shell scripts

#252
post #135

Earlier quoted context omitted.

BOFH much? It’s not as if this script is going to be used by people that have no idea what is going to happen. It’s a script, not a command. Your tone is very dismissive. Instead of criticism all of these could be phrased as suggestions instead. It’s like criticising your junior for being enthusiastic about everything they learned today.

I appreciate the parent comment and it's frankness. Not everyone, especially juniors, need to, or should be, coddled.

Sure, but there’s styles of writing and talking that have a counterproductive effect.

If your goal is improvement instead of venting you don’t want to use those.

Re: Techniques I use to create a great user experience for shell scripts

#253

Earlier quoted context omitted.

Thanks but I'm not really asking for advice. I'm uninterested in changing how I write correct, often POSIX-compliant shell scripts because of a linter that has an inferior understanding of the language. I'm also not a fan of this kind of dogmatic application of tools. Shellcheck can be useful sure, but my point is that, at least for me, the juice is often not worth the squeeze. I'm aware of how to disable rules. I of…

Shellcheck, and really any linter (and arguably also any other form of programming language safety, like static typing or compile-time memory safety), is not there for the very experienced author (which it sounds like you are). Those mechanisms exist for the inexperienced author (especially in a team setting) where you want some minimum quality and consistency. An example where Shellcheck might be useful for you is w…

Personally I disagree, but can understand your point.

I think I'm fairly experienced in shell and Python (~20 and ~8 YOE, respectively), and still find value in linters, type checkers, etc. Maybe moreso in Python, but that's probably a function of me writing larger programs in Python than in shell, and usually changing my mind on something as I'm writing it.

Re: Techniques I use to create a great user experience for shell scripts

#254

Earlier quoted context omitted.

Because I didn't see the edited version when I was writing my original reply and its too late, I want to call out another problem that you graciously overlooked that we can call #2 since it touches neatly on your #1 and #3 items and #2 is already missing. The extra junk you see in your wsl after the awk step is probably because the other big *NIX problem with shell scripts is my `netstat` or `grep` or even `echo` mig…

I was worried that my toy problem wasn’t complex enough to reveal these issues! I had an experience recently trying to deploy an agent on a dozen different Linux distros. I had the lightbulb moment that the only way to run IT in an org using exactly one distro. Ideally one version, two at the most during transitions. Linux is a kernel, not an operating system. There are many Linux operating systems that are only supe…

At least here, we can agree. If I ran a business and allowed employees to run Linux (which is reasonable, IMO), the last thing I want is someone's riced-out Gentoo with an unpatched security exploit onto the VPN.

Re: Techniques I use to create a great user experience for shell scripts

#255

If you want a great script user experience, I highly recommend avoiding the use of pipefail. It causes your script to die unexpectedly with no output. You can add traps and error handlers and try to dig out of PIPESTATUS the offending failed intermediate pipe just to tell the user why the program is exiting unexpectedly, but you can't resume code execution from where the exception happened. You're also now writing a…

> "It causes your script to die unexpectedly with no output." Oh? I don't observe this behavior in my testing. Could you share an example? AFAIK, if you don't capture stderr, that should be passed to the user. > "Instead, just check $? and..." I agree that careful error handling is ideal. However, IMO it's good defensive practice to start scripts with "-e" and pipefail. For many/most scripts, it's preferable to fail…

  $ date +%w
  0
  $ cat foo.sh 
  #!/usr/bin/env sh
  set -x
  set -eu -o pipefail
  echo "start of script"
  echo "start of pipe" | cat | false | cat | cat
  if [ "$(date +%w)" = "0" ] ; then
    echo "It's sunday! Here we do something important!"
  fi
  $ sh foo.sh
  + set -eu -o pipefail
  + echo 'start of script'
  start of script
  + echo 'start of pipe'
  + cat
  + false
  + cat
  + cat
  $
Notice how the script exits, and prints the last pipe it ran? It should have printed out the 'if ..' line next. It didn't, because the script exited with an error. But it didn't tell you that.

If you later find out the script has been failing, and find this output, you can guess the pipe failed (it doesn't actually say it failed), but you don't know what part of the pipe failed or why. And you only know this much because tracing was enabled.

If tracing is disabled (the default for most people), you would have only seen 'start of script' and then the program returning. Would have looked totally normal, and you'd be none the wiser unless whatever was running this script was also checking its return status and blaring a warning if it exited non-zero, and then you have an investigation to begin with no details.

> IMO it's good defensive practice to start scripts with "-e" and pipefail.

If by "defensive" you mean "creating unexpected failures and you won't know where in your script the failure happened or why", then I don't like defensive practice.

I cannot remember a single instance in 20 years where pipefail helped me. But plenty of times where I spent hours trying to figure out where a script was crashing and why, long after it had been crashing for weeks/months, unbeknownst to me. To be sure, there were reasons why the pipe failed, but in almost all cases it didn't matter, because either I got the output I needed or didn't.

> it's preferable to fail with inadequate output than to "succeed" but not perform the actions expected by the caller.

I can't disagree more. You can "succeed" and still detect problems and handle them or exit gracefully. Failing with no explanation just wastes everybody's time.

Furthermore, this is the kind of practice in backend and web development that keeps causing web apps to stop working, but the user gets no notification whatsoever, and so can't report an error, much less even know an error is happening. I've had this happen to me a half dozen times in the past month, from a bank's website, from a consumer goods company's website, even from a government website. Luckily I am a software engineer and know how to trace backend network calls, so I could discover what was going on; no normal user can do that.

Re: Techniques I use to create a great user experience for shell scripts

#256

Earlier quoted context omitted.

Sure, I'm not arguing that having a set of well defined outputs and passing objects around wouldn't be better. You're talking to someone that often laments that SmallTalk was not more popular. But you'd need to get the entire OSS community to land on a single object representation and then get them to independently change all the tools to start outputting the object version. PowerShell and Microsoft have the advantag…

Half way through your second paragraph I just knew you'd be reaching for 'jq'! All joking aside, that's not a bad solution to the underlying problem. Fundamentally, unstructured data in shell pipelines is much of the issue, and JSON can be used to provide that structure. I'm seeing more and more tools emit or accept JSON. If one can pinch their nose and ignore the performance overhead of repeatedly generating and par…

>Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell.

So this whole conversation and a different one about python and it's behavior around `exit` vs `exit()` sent me down a rabbit hole of seeing if I could make the python interpreter have a "shell like" dsl for piping around data. It turns out you sort of can. I don't think you can defeat the REPL and make a function call like `echo "foo" "bar" "baz", but you can make it do this:

  netstat("-ln") | filter_by({"state":"ESTABLISHED"}) \
    | group_by(["local_port"]) | count_groups \
    | sort("count", descending=True) | limit(10)
And only ever parse plain text once on the input from netstat. For your amusement/horror I present "ShPy": https://gitlab.com/tpmoney/shpy

Re: Techniques I use to create a great user experience for shell scripts

#257

Earlier quoted context omitted.

Shellcheck, and really any linter (and arguably also any other form of programming language safety, like static typing or compile-time memory safety), is not there for the very experienced author (which it sounds like you are). Those mechanisms exist for the inexperienced author (especially in a team setting) where you want some minimum quality and consistency. An example where Shellcheck might be useful for you is w…

Personally I disagree, but can understand your point. I think I'm fairly experienced in shell and Python (~20 and ~8 YOE, respectively), and still find value in linters, type checkers, etc. Maybe moreso in Python, but that's probably a function of me writing larger programs in Python than in shell, and usually changing my mind on something as I'm writing it.

I agree that there is value even for very experienced users. A good example of this is how expert C/C++ programmers still make mistakes with memory management -- memory safe languages benefits beginners and experts equally in this case.

I personally setup linters/formatters/other static analysis for solo projects, even for languages I know very well.

I just didn't want to write a comment large enough to capture all of the nuance :)

Re: Techniques I use to create a great user experience for shell scripts

#258

Earlier quoted context omitted.

I don’t think it’s fair to compare a workflow that is designed for sed/awk. It’s about 10 lines of python to run my command and capture stdout/stderr - the benefit of which is that I can actually read it. What happens if you want to retry a line if it fails?

> I don’t think it’s fair to compare a workflow that is designed for sed/awk. If your position is that we should not be writing bash but instead Python, then yes, it is absolutely fair. > the benefit of which is that I can actually read it. And you couldn't read the command pipeline I put together? > What happens if you want to retry a line if it fails? Put the thing you want to do in a function, execute it on a line…

My point is that if you take a snippet designed to be terse in bash, it’s an unfair advantage to bash. There are dozens of countless examples in python which will show the opposite

> And you couldn't read the command pipeline I put together?

It took me multiple goes, but the equivalent in python I can understand in one go.

> Put the thing you want to do in a function, execute it on a line, if the sub-shell returns a failure status, execute it again. It isn't like bash does not have if-statements or while-loops.

But when you do that, it all of a sudden looks a lot more like the python code

Re: Techniques I use to create a great user experience for shell scripts

#259
post #152

Earlier quoted context omitted.

What is better?

Xonsh. Been using it since 2018. Bash scripting sucks in comparison.

For reference, it seems to be this: https://xon.sh

  XONSH is a Python-powered shell

  Xonsh is a modern, full-featured and cross-platform shell. The language is a
  superset of Python 3.6+ with additional shell primitives that you are used to
  from Bash and IPython. It works on all major systems including Linux, OSX, and
  Windows. Xonsh is meant for the daily use of experts and novices.
Haven't heard of it before personally, and it looks like it might be interesting to try out.

Re: Techniques I use to create a great user experience for shell scripts

#260

Earlier quoted context omitted.

Thanks but I'm not really asking for advice. I'm uninterested in changing how I write correct, often POSIX-compliant shell scripts because of a linter that has an inferior understanding of the language. I'm also not a fan of this kind of dogmatic application of tools. Shellcheck can be useful sure, but my point is that, at least for me, the juice is often not worth the squeeze. I'm aware of how to disable rules. I of…

Shellcheck, and really any linter (and arguably also any other form of programming language safety, like static typing or compile-time memory safety), is not there for the very experienced author (which it sounds like you are). Those mechanisms exist for the inexperienced author (especially in a team setting) where you want some minimum quality and consistency. An example where Shellcheck might be useful for you is w…

This is one of the worst takes I've ever heard. People like you are the reason code breaks and kills people (or destroys property, etc.). Do you also refuse to use calculators, under the pretense of being too experienced, and as such calculating the square roots of four-digit numbers by hand?
Post reply on HN