Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to do things safely in Bash (2018)

github.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: How to do things safely in Bash (2018)

#91

Post by lenkite »

pwdisswordfish8 wrote: post #2

I've recently taken to not bothering with shell scripts at all. I just use Python instead. The type system, while it's by no means state of the art, is still miles ahead of the stringly-typed bash. The libraries are excellent; much can be done with the standard library alone. The scripts are much faster (even though the language isn't particularly tuned for performance), since I don't need to spawn a subprocess for e…

Use shell script for glueing stuff (ie calling other tools) and pipelines. Use GNU awk for complicated text processing. Profit!

Re: How to do things safely in Bash (2018)

#92

Post by chubot »

JNRowe wrote: post #64
chubot wrote: post #10

Earlier quoted context omitted.

Oil is Bourne compatible, but has a mode to opt you into the better behavior. Example: osh$ empty='' osh$ x='name with spaces.mp3' This is like Bourne shell: $ argv $empty $x ['name', 'with', 'spaces.mp3'] # omit empty and split $ argv "$empty" "$x" ['', 'name with spaces.mp3'] # unchanged Opt into better behavior, also available with bin/oil: $ shopt --set oil:basic $ argv $empty $x ['', 'name with spaces.mp3'] # no…

> Interestingly zsh also doesn't split words Should zsh users ever want that behavior they can enable it globally with the SH_WORD_SPLIT option, or more sensibly local to a given parameter with the = parameter expansion as in ${=var}. Also, it has the best comment in zsh's manpage: "SH_WORD_SPLIT [...] Note that this option has nothing to do with word splitting." $ x='name with spaces.mp3' $ print -l -- $=x # "print…

OK interesting, didn't know about those options. I have seen the ${(...)x} syntax but not really used it. Yes it doesn't rate highly on my readability scale :)

I have heard the feedback that people like zsh expansion shortcuts, e.g. for globbing. Personally I am a find/xargs person, i.e. I select the files first with 'find' and then execute what I want with xargs.

It does require you to invert your thinking -- you're going from verb NOUNS to NOUNS verb. And you have to go back to the beginning of the command line and edit it. But I do find that it lets you test out the selection logic more naturally.

find can also be faster, e.g.

    find . -name .git -a -prune -o -print
skips the even STATTING .git directory, not just printing it, as opposed to ** I believe.

However find arguably has an even worse syntax (although it is explicit, just with some dumb shortcuts). One longstanding goal is to put a better syntax on the "evaluation model", which is pretty useful. It's basically a predicate that's evaluated over every node in the FS tree, but you can also customize the traversal.

It might be better for programs rather than interactively, but I started using it interactively too.

(When compiled with glibc, Oil also has bash/ksh-style extended globbing, which allows negation etc., but it seems only old scripts use that.)

Re: How to do things safely in Bash (2018)

#93

Post by effie »

kaba0 wrote: post #46
effie wrote: post #29

Earlier quoted context omitted.

Indeed. Python does not win over bash in either category, and it is slower. I wish there was a super-fast shell-like scripting language everywhere that could serve as a general programming language. Like Perl, but simpler rules and readable syntax. But there is nothing. Who would have thought, I may have to learn Perl eventually.

How is it slower/matter at all? Goddamn bash calls an external process in if branches, so if a bash script is enough for a given process, than running python inside a vm inside a vm inside a vm will be still fast enough. Scripting doesn’t require performance.

It is slow because the process of interpretation is slow (conditions, number and string operations) and because to achieve anything nontrivial in shell/bash, one _has_ to call external processes, which is slow. In Perl one calls external processes much less because of large amount of capable libraries and Perl is indeed much faster.

Re: How to do things safely in Bash (2018)

#94

Post by effie »

drran wrote: post #37
effie wrote: post #29

Earlier quoted context omitted.

Indeed. Python does not win over bash in either category, and it is slower. I wish there was a super-fast shell-like scripting language everywhere that could serve as a general programming language. Like Perl, but simpler rules and readable syntax. But there is nothing. Who would have thought, I may have to learn Perl eventually.

IMHO, we need more than one language in the shell: one to orchestrate launching of applications, plumbing, and processing of their inputs and outputs, another one for computations with integers, floats, complex values, arrays, matrices, another one for text processing and parsing, another one for argument parsing and default values, another one for error handling and recovery, another one for pattern matching and sta…

Why not? Busybox is an example of how to pack diverse capabilities into one binary.

Re: How to do things safely in Bash (2018)

#95

Post by kaba0 »

effie wrote: post #93
kaba0 wrote: post #46

Earlier quoted context omitted.

How is it slower/matter at all? Goddamn bash calls an external process in if branches, so if a bash script is enough for a given process, than running python inside a vm inside a vm inside a vm will be still fast enough. Scripting doesn’t require performance.

It is slow because the process of interpretation is slow (conditions, number and string operations) and because to achieve anything nontrivial in shell/bash, one _has_ to call external processes, which is slow. In Perl one calls external processes much less because of large amount of capable libraries and Perl is indeed much faster.

This doesn’t make sense. How would python be slower? Bash is indeed the prototypical do what each line says. And there is absolutely no optimizations here.

Python while not the fastest thing out there, basically every implementation does a quick parse phase and run some optimized form.

Re: How to do things safely in Bash (2018)

#96

Post by effie »

kaba0 wrote: post #95
effie wrote: post #93

Earlier quoted context omitted.

It is slow because the process of interpretation is slow (conditions, number and string operations) and because to achieve anything nontrivial in shell/bash, one _has_ to call external processes, which is slow. In Perl one calls external processes much less because of large amount of capable libraries and Perl is indeed much faster.

This doesn’t make sense. How would python be slower? Bash is indeed the prototypical do what each line says. And there is absolutely no optimizations here. Python while not the fastest thing out there, basically every implementation does a quick parse phase and run some optimized form.

I meant bash is slower than perl. Python is also slower than perl. Python is quicker than bash in number/string operations in process, but much slower than bash in startup. 'Quick parse phase' is exactly the reason; it is not quick. Try to execute 1000 python processes and then 1000 bash processes that do something simple like add numbers given as command line arguments. Python is order of magnitude slower to finish.

Re: How to do things safely in Bash (2018)

#97

Post by LinuxBender »

stormbrew wrote: post #73
pmiller2 wrote: post #61

Earlier quoted context omitted.

So? Python 2.x is just as good for this, and is pretty widely available. Besides, I don't write scripts for every production machine in the world, just a subset that I have access to.

So, a bash script written ten years ago probably still works on any version of bash since then. A python script written ten years ago probably doesn't even parse on python3.0, let alone python3.6 (because there have been backwards incompatible changes since 3.0 even!). And that's also true vice versa. So like, this is to the point that there's some "reasonable subset" of python you can use to make it portable (ie. no…

Not too long ago I discovered that a tiny temporary shell script I gave to a team to validate Akamai cache was still being used nine plus years later. It was supposed to be a short term work around until the UI team created a proper validation. They never did and somehow my script kept working for almost a decade. The only reason I found out was due to a minor change in our UI that broke the manifest lookup url pattern.

Re: How to do things safely in Bash (2018)

#98

Post by JNRowe »

chubot wrote: post #92
JNRowe wrote: post #64

Earlier quoted context omitted.

> Interestingly zsh also doesn't split words Should zsh users ever want that behavior they can enable it globally with the SH_WORD_SPLIT option, or more sensibly local to a given parameter with the = parameter expansion as in ${=var}. Also, it has the best comment in zsh's manpage: "SH_WORD_SPLIT [...] Note that this option has nothing to do with word splitting." $ x='name with spaces.mp3' $ print -l -- $=x # "print…

OK interesting, didn't know about those options. I have seen the ${(...)x} syntax but not really used it. Yes it doesn't rate highly on my readability scale :) I have heard the feedback that people like zsh expansion shortcuts, e.g. for globbing. Personally I am a find/xargs person, i.e. I select the files first with 'find' and then execute what I want with xargs. It does require you to invert your thinking -- you're…

You don't have to choose glob&loop or find&xargs as a zsh user, as there is a built-in zargs function:

   zargs -- $crazy_glob -- $command
Much like the `find | xargs` version you can begin with `zargs -- $glob` until your filter is correct and then tack the command on when you're ready.

Hmm, now I really like the idea of an oil-y version of this where you could do something like `oargs { $clearer_glob_DSL } { $command_block }`. It might even be possible right now using stest from dmenu¹ as a stand-in for a more advanced globbing alternative.

I believe part of the reason zsh users find the extended globbing functionality useful is the basic usage matches their expectations with other tools, unlike find's quirky syntax. The common filters use the same values as you'd see in `ls --classify` output. For example, you want executable files tack a `*` on or for sockets use a `=`.

FWIW, the find comparison isn't quite right. Out of the box `**/file` wouldn't traverse a .git directory anyway, unless the GLOB_DOTS option is set or you provide the equivalent flag as in `**/file(D)`. The less specific point is quite true though as adding negation and toggling flags in a single glob can be extremely difficult to read. See the examples at the bottom of zshexpn(1) for proof of that.

Edit to add: I hope this comes across as an attempt to be helpful as was intended, and not some awful stop motion attempt.

¹ https://tools.suckless.org/dmenu/

Post reply on HN