Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use Long Options in Scripts

matklad.github.io

Post reply on HN
Print view View original

141–150 of 157 posts

Re: Use Long Options in Scripts

#141

Post by vsl »

Wowfunhappy wrote: post #87
susam wrote: post #84

I prefer long options too. However, while writing programs that need to invoke POSIX commands in a portable manner, short options are the only viable choice, as POSIX doesn't specify long options. For instance, see the specification for diff at https://pubs.opengroup.org/onlinepubs/9799919799/utilities/d... >, or that of any POSIX utility listed at https://pubs.opengroup.org/onlinepubs/9799919799/idx/utiliti... >. Th…

> However, while writing programs that need to invoke POSIX commands in a portable manner ...probably a stupid question, but something I have earnestly been wondering about... when does this actually happen nowadays? What POSIX systems are you targeting that aren't one of the major ones (Linux, Darwin, or one of the major BSDs)? I was writing a shell script a few months ago that I wanted to be very durable, and I tar…

Major ones are enough. Linux and Darwin (that is, macOS and GNU userspace, really) differ sufficiently that you need to pay attention or limit yourself to POSIX. E.g. sed and wc burned me a few times with scripts that need to run on both.

Re: Use Long Options in Scripts

#143

Post by chubot »

matklad wrote: post #86
wahern wrote: post #6

Please DO NOT mix string interpolation and command execution, especially when a command is processed through the shell. Whatever your language, use a list-based or array-based execution API that passes arguments straight through to execv(2), execvp(2), etc, bypassing the shell.

Was waiting for this comment :P The API used handles string interpolation correctly: the string literal is parsed at compile time, and the interpolated arguments are never concatenated or escaped, and end up directly as an element of arv array passed to a child. See https://github.com/tigerbeetle/tigerbeetle/blob/7053ecd2137a...

This approach creates an odd mini language, which is incomplete:

    comptime assert(std.mem.indexOfScalar(u8, cmd, '\'') == null); // Quoting isn't supported yet.
    comptime assert(std.mem.indexOfScalar(u8, cmd, '"') == null);
But you can do correct interpolation with simple shell variables, rather than generating shell code strings:

    $ today='foo; bar' sh -c 'argv git switch --create "release-$today" origin/main'
    ['git', 'switch', '--create', 'release-foo; bar', 'origin/main']
So that is a test that we can use a plain shell string, without any shell injection bug. (argv is my command to test quoting: python -c 'import sys; print(sys.argv)' "$@" )

Note that there's no escaping function needed, because we're not generating any shell code. We're generating an argv array for `/bin/sh` instead.

---

So by invoking with an env var, you can easily create a correct API that uses plain shell

    git switch --create "release-$today"
rather than

    git switch --create release-{today}  # what language is this?  It's not obvious
If you don't want to use the env var, you can also use

    git switch --create "release-$1"
And invoke with

    ['sh', '-c', shell_string, 'unused-arg0', today_string]
With this approach, you don't need

    1. any kind of shell escaping
    2. any analyzing of pseudo-shell strings, which can't contain quotes
Because you are not generating any shell code. The shell code is constant.

Re: Use Long Options in Scripts

#144

Post by gorgoiler »

kbenson wrote: post #138
gorgoiler wrote: post #137

Earlier quoted context omitted.

I don’t really follow. My “good” example and the code at the bottom are the same. sh is smarter than just doing string interpolation and ”$1” is passed on as a single argument, no matter what: > run(["sh", "-c", 'echo "$1"', "--", 'a"']) a” Whereas if it were simple string interpolation, you’d see this: > run(["sh", "-c", 'echo "a""') --: 1: Syntax error: Unterminated quoted string It’s the same special casing that g…

That requires you quote the param in the sintr to ensure that params are groups as expected. E.g. # cat pvars.sh #!/bin/bash echo "\$1=$1" echo "\$2=$2" echo "\$3=$3" # sh -c './pvars.sh "$1" $2 $3' -- 'a b' 'c d' $1=a b $2=c $3=d The whole point of passing in an array and using something like exec (or system(), if provided as it handled the fork and wait for you) is that you avoid the overhead of the shell starting…

Thanks for explaining. I feel like we’re talking past each other but it’s my mistake. I should have said it is only useful (not “particularly useful”) if one has compound statements like a pipe or multiple commands. Invoking sh just to run a single command is superfluous and you are right that reaching directly for execve() is better.

Re: Use Long Options in Scripts

#145

Post by wodenokoto »

mhitza wrote: post #95
amelius wrote: post #88

Before invoking a command, always first check if the length of the command is not longer than ARG_MAX. For example, if this is your command: grep --ignore-case --files-with-matches -- "hello" *.c Then invoke it as follows: CMD="grep --ignore-case --files-with-matches -- \"hello\" *.c" ARG_MAX=$(getconf ARG_MAX) CMD_LEN=${#CMD} if (( CMD_LEN > ARG_MAX )); then echo "Error: Command length ($CMD_LEN) exceeds ARG_MAX ($A…

That might be sensible, but also obscure the script logic. Since using Linux exclusively, I don't think I've ever encountered an issue due to too many arguments/length. And it's the first time I'm actively searching online for ARG_MAX. I understand that different shells might be different, but with reasonable lengths is there any chance of it being relevant (aside from xargs, where it's generally intended, or better,…

I've had problems with "cat *.csv" plenty of times processing data that is generated in many small files.

It is really difficult to deal with, because on top of the arg max limit, globs are not guaranteed to be in order.

The solution is not obvious and hard to get to if you don't know the foot guns in advanced and hard to read once implemented.

Re: Use Long Options in Scripts

#147

Post by [deleted] »

Timon3 wrote: post #105
apgwoz wrote: post #96

Earlier quoted context omitted.

You should always type check your shell scripts as well. For example, you just: $ shelltypes script.sh # Welcome to shelltypes v 3.23.2 # type ‘help’ if you’re stuck >>> {1) # import POSIX;; Importing 73 items. >>> {2} # append loadpath “/opt/local/shelltypes/base”;; >>> {3} # import base::YOURPROJECT;; Importing 15 items. >>> {4} # check “YOURSCRIPT.sh” Parsing YOURSCRIPT.sh. Reticulating splines. Expanding aliases.…

Where does "shelltypes" come from? I can't find anything on DuckDuckGo or Google, but this seems like it would be very useful.

[deleted]

Re: Use Long Options in Scripts

#148

Post by em500 »

Wowfunhappy wrote: post #87
susam wrote: post #84

I prefer long options too. However, while writing programs that need to invoke POSIX commands in a portable manner, short options are the only viable choice, as POSIX doesn't specify long options. For instance, see the specification for diff at https://pubs.opengroup.org/onlinepubs/9799919799/utilities/d... >, or that of any POSIX utility listed at https://pubs.opengroup.org/onlinepubs/9799919799/idx/utiliti... >. Th…

> However, while writing programs that need to invoke POSIX commands in a portable manner ...probably a stupid question, but something I have earnestly been wondering about... when does this actually happen nowadays? What POSIX systems are you targeting that aren't one of the major ones (Linux, Darwin, or one of the major BSDs)? I was writing a shell script a few months ago that I wanted to be very durable, and I tar…

> ... but I don't actually know what system in the past decade (or more) wouldn't have bash.

There's some ambiguity about "have bash". If "having" bash means that (some version of) bash has been ported to the system, there are indeed very few. If "having" means that bash (supporting all options that you need) is available to the user, that could be a lot more. As others have noted, the BSDs, Android and many embedded Linux systems don't come with bash pre-installed, MacOS pre-installed bash is stuck at version 3.2 (which doesn't have associative arrays), and the user could be in an environment that does not allow them to install whatever they need.

Re: Use Long Options in Scripts

#149

Post by kbenson »

gorgoiler wrote: post #144
kbenson wrote: post #138

Earlier quoted context omitted.

That requires you quote the param in the sintr to ensure that params are groups as expected. E.g. # cat pvars.sh #!/bin/bash echo "\$1=$1" echo "\$2=$2" echo "\$3=$3" # sh -c './pvars.sh "$1" $2 $3' -- 'a b' 'c d' $1=a b $2=c $3=d The whole point of passing in an array and using something like exec (or system(), if provided as it handled the fork and wait for you) is that you avoid the overhead of the shell starting…

Thanks for explaining. I feel like we’re talking past each other but it’s my mistake. I should have said it is only useful (not “ particularly useful”) if one has compound statements like a pipe or multiple commands. Invoking sh just to run a single command is superfluous and you are right that reaching directly for execve() is better.

Ah, yes. If you want to take advantage of piping commands together through the shell as a subcommand of your program, then a way to make params behave more consistently regardless of content is useful.
Post reply on HN