Live data from Hacker News

Techniques I use to create a great user experience for shell scripts

nochlin.com

181–190 of 281 posts

Re: Techniques I use to create a great user experience for shell scripts

#181

Earlier quoted context omitted.

You could be right - I certainly didn't intend my comment to be antagonistic. My experience of ShellCheck is that you only get loads of warnings when you first start out using it and it finds all of your unquoted variables. Once you get more experienced with writing scripts and linting them with ShellCheck, the number of warnings should dramatically reduce, so it seems odd that an experienced script writer would be f…

> it seems odd that an experienced script writer would be falling foul of shellcheck being pedantic about what you're writing. It's not simply being pedantic, it is wrong . Your writing gives the impression that the tool is infallible. If I was new to writing shell scripts, shellcheck is clearly a wise choice. The language is loaded with footguns. But as someone who has been writing scripts for decades, I already kno…

I do agree about ShellCheck being wrong sometimes - that's why I mentioned the method of disabling it for specific lines with a comment. When I first started using ShellCheck, it was highlighting lots of footguns that I wasn't aware of, but nowadays, it's very rare for it to spit a warning out at me - usually just for something like it not following a script or stating that a variable wasn't defined when it was.

I think the huge number of footguns is what makes BASH scripting fun.

Re: Techniques I use to create a great user experience for shell scripts

#182
post #80

Earlier quoted context omitted.

For fun, I took a crack at your example and came up with this craziness (with the caveat it's late and I didn't spend much time on it), which is made a bit more awkward because grep doesn't do capturing groups: netstat -aln \ | grep ESTABLISHED \ | awk '{print $4}' \ | grep -Po '\:\d+$' \ | grep -Po '\d+' \ | sort \ | uniq -c \ | sort -r \ | head -n 10 Changing the awk field to 5 instead of 4 should get you remote po…

I know this sounds like nit-picking but bear with me. It's the point I'm trying to make: 1. Your script outputs an error when run, because 'bash' itself doesn't have netstat as a built-in. That's an external command. In my WSL2, I had to install it. You can't declaratively require this up-front, you script has to have an explicit check... or it'll just fail half-way through. Or do nothing. Or who knows!? PowerShell h…

> PowerShell has up-front required prerequisites that you can declare

Anyone who's written more than a few scripts for others will have learned to do something like this at the start:

    declare -a reqs
    reqs+=(foo bar baz)
    missing=0
    for r in "${reqs[@]}"; do
        if (! command -v "$r" &>/dev/null); then
            echo "${r} is required, please install it"
            missing=1
        fi
    done
    if [ $missing -gt 0 ]; then
        exit 1
    fi
> Your script is very bravely trying to parse output that includes many different protocols, including: tcp, tcp6, udp, udp6, and unix domain sockets

They probably didn't know you could specify a type. Mine only displays TCP4.

> Now imagine putting your script side-by-side with the PowerShell script, and giving it to people to read.

I'm gonna gatekeep here. If you don't know what that script would do, you have no business administering Linux for pay. I'm not saying that in a "GTFO noob" way, but in a "maybe you should know how to use your job's tools before people depend on you to do so." None of that script is using exotic syntax.

> Note that you had to use 'awk', which is a parser, and then three uses of 'grep' -- a regular expression language, which is also a kind of parsing.

They _chose_ to. You _can_ do it all with awk (see my example in a separate post).

> Literally in every discussion about PowerShell there's some Linux person who's only ever used bash complaining that PS syntax is "weird" or "hard to read". What are they talking about!? It's half the complexity for the same functionality, reads like English, and doesn't need write-only hieroglyphics for parameters.

And yet somehow, bash and its kin continue to absolutely dominate usage.

There is a reason that tools like ripgrep [0] are beloved and readily accepted: they don't require much in the way of learning new syntax; they just do the same job, but faster. You can load your local machine up with all kinds of newer, friendlier tools like fd [1], fzf[2], etc. – I definitely love fzf to death. But you'd better know how to get along without them, because when you're ssh'd onto a server, or god forbid, exec'd into a container built with who-knows-what, you won't have them.

Actually, that last point sparked a memory: what do you do when you're trying to debug a container and it doesn't have things like `ps` available? You iterate through the `/proc` filesystem, because _everything is a file._ THAT is why the *nix way exists, is wonderful, and is unlikely to ever change. There is always a way to get the information you need, even if it's more painful.

[0]: https://github.com/BurntSushi/ripgrep

[1]: https://github.com/sharkdp/fd

[2]: https://github.com/junegunn/fzf

Re: Techniques I use to create a great user experience for shell scripts

#183
post #35

Earlier quoted context omitted.

I also hate bash scripting, and as far as Unix shell go, bash is among the best. So many footguns... Dealing with filenames with spaces is a pain, and files that start with a '-', "rm -rf" in a script is a disaster waiting to happen unless you triple check everything (empty strings, are you in the correct directory, etc...), globs that don't match anything, etc... But interactively, I much prefer Unix shells over Pow…

> What you are saying essentially is that PowerShell is a better programming language than bash In some sense, yes, but there is no distinct boundary. Or at least, there ought not to be one! A criticism a lot of people (including me) had of Windows in the NT4 and 2000 days was that there was an enormous gap between click-ops and heavyweight automation using C++ and COM objects (or even VBScript or VB6 for that matter…

Windows still has horrendous automation support, PowerShell falls short and loses its USP as soon as you need anything that is not a builtin and series of bandaids like DSC didn't even ameliorate the situation. The UX is bad even when working with nothing but MS products like MSSQL.

The biggest leap for automation on Windows has been WSL, aka shipping Linux.

Re: Techniques I use to create a great user experience for shell scripts

#184
post #122

Earlier quoted context omitted.

> shell scripts are the wrong solution for anything over ~50 lines of code. I don't think LOC is the correct criterion. I do solve many problems with bash and I enjoy the simplicity of shell coding. I even have long bash scripts. But I do agree that shell scripting is the right solution only if = you can solve the problem quickly = you don't need data structures = you don't need math = you don't need concurrency

I enjoy the simplicity of shell coding You mean, the complexity of shell coding? Any operation that in a regular language is like foo.method(arg) in shell expands into something like ${foo#/&$arg#%} or `tool1 \`tool2 "${foo}"\` bar | xargs -0 baz`.

> in a regular language [...] like foo.method(arg)

Note what you just said: when you want an object with a method that takes a parameter, you find bash too complex.

You gave an example that is not appropriate for bash.

However, bash does have functions. So if you don't need an entire underlying object system just to run your logic, you could have

    function foomethod () { 
        parm1=$1
        #more logic   
    }

Re: Techniques I use to create a great user experience for shell scripts

#185

Earlier quoted context omitted.

Sure, I'm not arguing that having a set of well defined outputs and passing objects around wouldn't be better. You're talking to someone that often laments that SmallTalk was not more popular. But you'd need to get the entire OSS community to land on a single object representation and then get them to independently change all the tools to start outputting the object version. PowerShell and Microsoft have the advantag…

Half way through your second paragraph I just knew you'd be reaching for 'jq'! All joking aside, that's not a bad solution to the underlying problem. Fundamentally, unstructured data in shell pipelines is much of the issue, and JSON can be used to provide that structure. I'm seeing more and more tools emit or accept JSON. If one can pinch their nose and ignore the performance overhead of repeatedly generating and par…

>Years ago, a project idea I was really interested for a while was to try to write a shell in Rust that works more like PowerShell. >Where I got stuck was the fundamentals: PowerShell heavily leans on the managed virtual machine and the shared memory space and typed objects that enables.

Hmmm, if you need that sort of shared memory access throughout the shell, you probably need a language like python (or maybe better Lisp) with a REPL and the ability/intent to self modify while running. Of course, every time you have to farm out because you don't have a re-written replacement internally to the shell app, you'd still parsing strings, but at least you could write a huge part of data processing in the shell language and keep it in house. Years ago I worked for a company that was using Microsoft's TFVC services (before it was azure devops or whatever they call it now) and wrote a frontend to their REST API in python that we could call from various other scripts and not be parsing JSON everywhere. Where this is relevant to the discussion is that one of the things I built in (in part to help with debugging when things went sideways) was an ability to drop into the python REPL mid-program run to poke around at objects and modify them or the various REST calls at will. With well defined functions and well defined objects , the interactive mode was effectively a shell for TFVC and the things we were using.

Though all of that said, even if one did that, they would still either need to solve the "object model" problem for disparate linux tools, or worse commit to writing (or convincing other people to write and maintain) versions of all sorts of various tools in the chosen language to replace the ones the shell isn't farming out to anymore. Its one thing to chose to write a shell, it's something else entirely to choose to re-write the gnu userland tools (and add tools too)

Re: Techniques I use to create a great user experience for shell scripts

#186
post #140

Earlier quoted context omitted.

I think a lot of people jump to Go because it’s nearly as convenient as bash for writing shell scripts?

People say that (and the same for Python), but I just don’t get it – and I’m a huge fan of Python. With shell, I can take the same tools I’ve already been using as one-liners while fiddling around, and reuse them. There is no syntax mapping to do in my head. With any other language, I have to map the steps, and probably also add various modules (likely within stdlib, but still). That’s not nothing. I’ve rewritten a s…

Completely agree.

I have a rule that if I’m using arrays I should move to Python, PHP, etc. It’s a nice red flag. Arrays in Bash are terrible and a sign that things are getting more complicated.

Re: Techniques I use to create a great user experience for shell scripts

#187

Earlier quoted context omitted.

> * #!/bin/bash instead of #!/usr/bin/env bash Except that'll pick up an old (2006!) (unsupported, I'm guessing) version of bash (3.2.57) on my macbook rather than the useful version (5.2.26) installed by homebrew. > -z instead of actually checking how many arguments you got I think that's fine here, though? It's specifically wanting the first argument to be a non-empty string to be interpolated into a filename later…

> Except that'll pick up an old (2006!) (unsupported, I'm guessing) version of bash (3.2.57) on my macbook rather than the useful version (5.2.26) installed by homebrew. Could you change that by amending your $PATH so that you're preferred version is chosen ahead of the default?

> Could you change that by amending your $PATH

I think the `#!/bin/bash` will always invoke that direct file without searching your $PATH. People say you can do `#!bash` to do a $PATH search but I've just tried that on macOS 15 and an Arch box running a 6.10.3 kernel and neither worked.

Re: Techniques I use to create a great user experience for shell scripts

#188

Earlier quoted context omitted.

Addendum after reading the script: * #!/bin/bash instead of #!/usr/bin/env bash * [ instead of [[ * -z instead of actually checking how many arguments you got passed and trusting the end user if they do something weird like pass an empty string to your program * echo instead of printf * `print_and_execute sdk install java $DEFAULT_JAVA_VERSION` who asked you to install things? * `grep -h "^sdk use" "./prepare_$fork.s…

> * #!/bin/bash instead of #!/usr/bin/env bash Except that'll pick up an old (2006!) (unsupported, I'm guessing) version of bash (3.2.57) on my macbook rather than the useful version (5.2.26) installed by homebrew. > -z instead of actually checking how many arguments you got I think that's fine here, though? It's specifically wanting the first argument to be a non-empty string to be interpolated into a filename later…

The 1brc shell script uses `#!/bin/bash` instead of `#!/usr/bin/env bash`. Using `#!/usr/bin/env bash` is the only safe way to pick up a `bash` that’s in your $PATH before `/usr/bin`. (You could do `#! bash`, but that way lies madness.)

Re: Techniques I use to create a great user experience for shell scripts

#189
post #152
post #136

Earlier quoted context omitted.

As a bash casual, these suggestions are a reminder of why I avoid using bash when I can. That's a whole armory of footguns right there.

What is better?

Xonsh. Been using it since 2018. Bash scripting sucks in comparison.

Re: Techniques I use to create a great user experience for shell scripts

#190
post #80

Earlier quoted context omitted.

For fun, I took a crack at your example and came up with this craziness (with the caveat it's late and I didn't spend much time on it), which is made a bit more awkward because grep doesn't do capturing groups: netstat -aln \ | grep ESTABLISHED \ | awk '{print $4}' \ | grep -Po '\:\d+$' \ | grep -Po '\d+' \ | sort \ | uniq -c \ | sort -r \ | head -n 10 Changing the awk field to 5 instead of 4 should get you remote po…

I know this sounds like nit-picking but bear with me. It's the point I'm trying to make: 1. Your script outputs an error when run, because 'bash' itself doesn't have netstat as a built-in. That's an external command. In my WSL2, I had to install it. You can't declaratively require this up-front, you script has to have an explicit check... or it'll just fail half-way through. Or do nothing. Or who knows!? PowerShell h…

Because I didn't see the edited version when I was writing my original reply and its too late, I want to call out another problem that you graciously overlooked that we can call #2 since it touches neatly on your #1 and #3 items and #2 is already missing. The extra junk you see in your wsl after the awk step is probably because the other big *NIX problem with shell scripts is my `netstat` or `grep` or even `echo` might not be the same as yours. I originally wrote it on a mac, and while I was checking the man page for netstat to see how old it was and how likely netstat output would change, it occurred to me that BSD netstat and linux netstat are probably different, so I jumped over and re-wrote on a linux box. Entirely possible your version is different from mine.

Heck, just checking between Bash, ZSH and Fish on my local machine here and Bash and ZSH's version is from 2003 and provides a single `-n` option and declares POSIX compliance, but explicitly calls out that `sh`'s version doesn't accept the `-n` argument. Fish provides their own implementation that accepts arguments `[nsEe]` from 2023. Every day I consider it a miracle that most of the wider internet and linux/unix world that underlies so much of it works at all, let alone reliably enough to have multiple nines of uptime. "Worse is better" writ large I guess.

Post reply on HN