Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I Wish I'd Known About Bash

zwischenzugs.com

Post reply on HN
Print view View original

151–160 of 272 posts

Re: Things I Wish I'd Known About Bash

#151

Post by tzahola »

lobster_johnson wrote: post #141
mdaniel wrote: post #138

Earlier quoted context omitted.

pipefail do nothing here Isn't that because the syntax provided doesn't use a pipe? Which is why actually using a pipe, rather than paren redirection, is easier on the eyes and the shell. It reads more like the flow of text: maybe_failing() { if ! some-failing-command; then echo "That did not shake out" >&2 return 1 fi } maybe_failing | some-command Is that one of the things StackOverflow said? And if so, why is it n…

This requires that some-command takes input from stdin (many commands don't) and that there's only a single input. What about: process-stuff --file This is the point at which one reaches for tempfiles, usually. The point remains that Edit: Isn't substitution using pipes, though? As in FIFO pipes? You get a file descriptor device which is closed at the end of the script. I don't know if the fd is wired directly to the…

>I don't know if the fd is wired directly to the command's stdout or whether the output is written in its entirety to a hidden tempfile first; the former sounds more natural and efficient.

Nope. () is equivalent to creating a named pipe with mkfifo and writing to it:

    tmp_pipe_dir="$(mktemp -d)"
    mkfifo "$tmp_pipe_dir/pipe"
    make-input >"$tmp_pipe_dir/pipe" &
    process-stuff --file "$tmp_pipe_dir/pipe"
That's the whole point of it, so that the data can be consumed in parallel with the producer.

Re: Things I Wish I'd Known About Bash

#152

Post by gumby »

gumby wrote: post #43

If you want all the args to the previous (or earlier) command just use ! . (e.g. a common idiom for me is cat `! `). Or you tried a git mv out of habit but the dir isn’t being managed by git: !gi:

Oops that’s the asterisk (star character) that HN interpreted as italics. In hacker news formattingnmy two examples are

  `!*` and !gi:*

Re: Things I Wish I'd Known About Bash

#153

Post by lobster_johnson »

tzahola wrote: post #151
lobster_johnson wrote: post #141

Earlier quoted context omitted.

This requires that some-command takes input from stdin (many commands don't) and that there's only a single input. What about: process-stuff --file This is the point at which one reaches for tempfiles, usually. The point remains that Edit: Isn't substitution using pipes, though? As in FIFO pipes? You get a file descriptor device which is closed at the end of the script. I don't know if the fd is wired directly to the…

>I don't know if the fd is wired directly to the command's stdout or whether the output is written in its entirety to a hidden tempfile first; the former sounds more natural and efficient. Nope. () is equivalent to creating a named pipe with mkfifo and writing to it: tmp_pipe_dir="$(mktemp -d)" mkfifo "$tmp_pipe_dir/pipe" make-input >"$tmp_pipe_dir/pipe" & process-stuff --file "$tmp_pipe_dir/pipe" That's the whole po…

Right, that's what I meant. The command is run in a child process with its stdout writing to the pipe. The script gets the read end.

Re: Things I Wish I'd Known About Bash

#154

Post by mdaniel »

lobster_johnson wrote: post #141
mdaniel wrote: post #138

Earlier quoted context omitted.

pipefail do nothing here Isn't that because the syntax provided doesn't use a pipe? Which is why actually using a pipe, rather than paren redirection, is easier on the eyes and the shell. It reads more like the flow of text: maybe_failing() { if ! some-failing-command; then echo "That did not shake out" >&2 return 1 fi } maybe_failing | some-command Is that one of the things StackOverflow said? And if so, why is it n…

This requires that some-command takes input from stdin (many commands don't) and that there's only a single input. What about: process-stuff --file This is the point at which one reaches for tempfiles, usually. The point remains that Edit: Isn't substitution using pipes, though? As in FIFO pipes? You get a file descriptor device which is closed at the end of the script. I don't know if the fd is wired directly to the…

Yes, you absolutely have me about the stdin part

But, the multiple case example highlights that if (for argument's sake) `make-input` and `make-more-stuff` had run _prior_ to that line, writing their (possibly empty) output to a (file|FIFO), how then would you want bash to behave? It would still open file descriptors to those (file|FIFO)s, which would still be just as blank.

It seems to me that if one wishes more fine grained control over the error handling in a multi-subshell-fd-trickery situation, then creating the FIFO(s) and managing the sender's exit status is the supervising script's responsibility.

a file descriptor device which is closed at the end of the script

I checked, and it's actually not even at the end of the script; those fds only exist for that one child, as `process-stuff` is exec-ed by bash.

whether the output is written in its entirety to a hidden tempfile

It's not a temp-file, it's an actual file descriptor, which bash `dup2`s for the subprocess, then cheekily uses the `/dev/fd/63` syntax to make appear as a file; you can peer into its brain a little:

    $ showme() {
        echo "showme.args=$@" >&2
        the_fd="${1##/dev/fd/}"
        cat
Thanks so much for highlighting this scenario; I learned a ton about how that works researching this answer. That's why I like answering stuff on S.O., too: win-win

Re: Things I Wish I'd Known About Bash

#156

Post by lobster_johnson »

mdaniel wrote: post #154
lobster_johnson wrote: post #141

Earlier quoted context omitted.

This requires that some-command takes input from stdin (many commands don't) and that there's only a single input. What about: process-stuff --file This is the point at which one reaches for tempfiles, usually. The point remains that Edit: Isn't substitution using pipes, though? As in FIFO pipes? You get a file descriptor device which is closed at the end of the script. I don't know if the fd is wired directly to the…

Yes, you absolutely have me about the stdin part But, the multiple case example highlights that if (for argument's sake) `make-input` and `make-more-stuff` had run _prior_ to that line, writing their (possibly empty) output to a (file|FIFO), how then would you want bash to behave? It would still open file descriptors to those (file|FIFO)s, which would still be just as blank. It seems to me that if one wishes more fin…

Good point about how to abort. Maybe it would be possible to short-circuit the fd somehow, so that the reader got an EOF, and that would in turn cause failure in the program reading the stream. At the same time, any success code from the program should be turned into an error exit code so the script could detect the failure. Not perfect, but arguably better than no failure at all.

Re: Things I Wish I'd Known About Bash

#157

Post by itwy »

joshbaptiste wrote: post #17

Hang out in #bash on IRC Freenode and you will be a Bash jedi http://mywiki.wooledge.org/BashFAQ best resource IMO for quick Bash syntax lookups as I always need to refer to the BashFaq to remember parameter expansion sub-string retrieval. parameter result ----------- ------------------------------ $name polish.ostrich.racing.champion ${name#*.} ostrich.racing.champion ${name##*.} champion ${name%%.*} polish ${name%.…

They are needlessly rude and mean at #bash. A bunch of scumbags, actually.

Re: Things I Wish I'd Known About Bash

#158

Post by joncrane »

sethrin wrote: post #30

I wrote a book on Bash too. The most important thing for anyone to know about Bash is that it's intended as a command language, not a general purpose scripting language. If it's longer than 10 lines, or if it uses two or more variables, you should probably have written it in something other than Bash.

How do you feel about the use of bash for AWS userdata and the like?

You pretty much have to put all the init stuff in a bash script. I can't think of any real-world examples of userdata scripts being less than 10 lines.

The alternative of course would be a three line script that downloads a file and executes it. But what language would that file be in? Probably bash. And it would make the infrastructure-as-code tracking and deployment process much more complicated.

Re: Things I Wish I'd Known About Bash

#159

Post by DonHopkins »

kinkrtyavimoodh wrote: post #128
pritambaral wrote: post #92

Earlier quoted context omitted.

As someone who also has to semi-frequently modify 100+ lines bash scripts written by others, I'd suggest every serious bash scripter read the bash man page. It is much smaller than any book on Python. For scripts written in Python, I'd use a similar argument and suggest every serious Python scripter to learn Python. As for Perl, or Ruby, or Julia, or anything really. It's just that learning bash from its man page is,…

I agree. And I agree with your bash 'whitelist'. I'd also add a hard blacklist for any serious string manipulation. A series of awks and seds look clever but they are quite annoying to deal with. If it's just a bunch of cuts or tr , sure.

And it's SO MUCH MORE expensive to fork all those processes. Just use a real language, please.

Re: Things I Wish I'd Known About Bash

#160

Post by [deleted] »

erk__ wrote: post #125
jwilk wrote: post #116

$ grep somestring file1 > /tmp/a $ grep somestring file2 > /tmp/b $ diff /tmp/a /tmp/b You shouldn't do that, but not because it's not neat enough. /tmp is world-writable, so you might be writing to somebody else's file, or over a symlink that was set up by someone else. Use mktemp¹ for creating temporary files. ¹ http://man7.org/linux/man-pages/man1/mktemp.1.html

Could you not do that with pipes instead, something like: $ diff

[deleted]
Post reply on HN