Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bash Pitfalls

bash.cumulonim.biz

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Bash Pitfalls

#31

Post by sigil »

joseph wrote: post #28
sigil wrote: post #20

Earlier quoted context omitted.

> So does that mean, that "for" will do something per word of the output of $, rather than per line of output of it? Correct. The argument to "for" is a list of words. > What to do if I want to do something for every line? Use a while loop. find /some/dir/ -type f | while read -r line; do ; # something with $line done PS. You should almost always use `find` instead of `ls` in shell scripts. Given a pattern, `ls` will…

One thing to be careful of when doing "while read..." is that a new shell is started on each iteration, so you cannot for example set a variable within the loop that you can use later in the script, as its value will be lost when the shell process exits.

> a new shell is started on each iteration

This is not actually true.

    printf "\n\n\n" | while read i; do a="x$a"; echo "$a"; done
    x
    xx
    xxx
The accumulator value even carries over after the while loop:

    printf "\n\n\n" | ( while read i; do a="x$a"; echo "$a"; done ; echo "$a" )
    x
    xx
    xxx
    xxx
(Technically, whether or not the loop body is executed in a subshell may be implementation dependent. Haven't looked at the POSIX shell spec in a while, but I seem to remember an old ksh that actually used subshells. At any rate, none of the modern sh's and bash force a subshell.)

What is true, however, is that a pipeline will execute in a subshell. Maybe that's what you're getting at here, and it is an important caveat.

    a=y; printf "\n\n\n" | while read i; do a="x$a"; echo "$a"; done; echo "$a"
    xy
    xxy
    xxxy
    y

Re: Bash Pitfalls

#32

Post by sigil »

brennen wrote: post #26
brandonbloom wrote: post #12

Maybe some controversial advice: Go ahead, fall in these pits. I write my fair share of shell scripts and I've hit practically every one of these snags in the past. However, for the majority of tasks I perform with bash, I genuinely don't care if I support spaces in filenames, or if I throw away a little efficiency with a few extra sub-shells, or if I can't test numbers vs strings or have a weird notion of booleans.…

I live and die by the shell. I'm constantly composing little one-liners, and keep an absurdly long Bash / zsh history to draw from. There are places the obvious answer is almost always "how about you just write a shell script?" That said, I long ago reached a place where I realized that, while shell scripting is entertaining , I'd much rather write anything more than a handful of lines in a general purpose programmin…

> Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives.

I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?"

Statefulness itself is a pain, and shell is in some sense the ultimate language for simply and flexibly dealing with external state.

Simplicity: the filesystem is an extremly simple and powerful state representation. Show me a language that interacts with the fs more concisely than

    tr '[A-Z]' '[a-z]'  lower.txt
Flexibility: if shell can't do it, just use another program in another language that can, like `tr` in the above example. What other language enables polyglot programming like this? Literally any program in any language can become a part of a shell program.

> it's just that once you're past a certain very low threshold of complexity, the agony you spend for a piece of reusable code is so much less.

Here's where I admit I was playing devil's advocate to an extent, because I fully agree with you here. I write lots of shell scripts. I never write big shell scripts. Above some length they just get targeted for replacement in a "real" language, or at the very least, portions of them get rewritten so they can remain small.

Empirically, it also seems true that shell is harder for people to grasp, harder to read, and harder for people to get right. These are real costs that have to be figured in.

PS. Speaking of shell brennen, we should be working on our weekend project. :)

Re: Bash Pitfalls

#33

Post by anon4 »

I know someone will sooner or later propose that we ban spaces and special characters in names. Let me just put my two cents forward.

We should absolutely ban special characters from names. Specifically, all whitespace, the colon, semicolon, forward slash, backward slash, question mark, star, ampersand, and whatever else I'm missing that will confuse the shell. Also files cannot start with a dash.

However, people should be able to name files with these characters. So I propose that these characters in filenames be percent-encoded like they would be in a URL. Specifically, the algorithm should be

1. Take the file name and encode it as UTF-8. Enforce some sort of normalization.

2. Substitute each problematic byte with equivalent percent-encoded form. This does not touch bytes over 0x80 - they are assumed non-problematic.

3. Write the file in the file system under that name.

4. When displaying files, run the algorithm in reverse.

In the general case files like "01 - Don't Eat the Yellow Snow.mp3" would simply become 01%20-%20Don't%20Eat%20the%20Yellow%20Snow.mp3 in the filesystem and cause absolutely no further problems. To make it completely backwards-compatible we should also add the following rule: If a filename includes a problematic byte or a percent-encoded byte higher than 0x80, then it is assumed to be raw and will not undergo percent decoding.

Basically, I propose that every program which receives free text input for a file name percent-encode the filenames before writing them to the filesystem and decode them for display. Everything else remains unchanged.

Why this will not work:

Requiring programmers to keep track of two filenames instead of just one is rather a lot of work. File APIs will have to take both encoded and non-encoded forms and encode the non-encoded form, creating problems when people inadvertently use the wrong function with a name, either double-encoding it or not encoding it and leading to "this file does not exist" errors.

It will be possible to create two files with different names on disk which are nonetheless shown with the same name to the user.

Why it is ugly:

We're taping over a deficiency of an ancient language by inflicting pain on programmers.

Double-encoded filenames? MADNESS.

Why I like it:

I'll be able to have ?, * and : in filenames in windows.

My shell scripts will be much simpler.

What do you guys think?

Re: Bash Pitfalls

#34

Post by comex »

sigil wrote: post #32
brennen wrote: post #26

Earlier quoted context omitted.

I live and die by the shell. I'm constantly composing little one-liners, and keep an absurdly long Bash / zsh history to draw from. There are places the obvious answer is almost always "how about you just write a shell script?" That said, I long ago reached a place where I realized that, while shell scripting is entertaining , I'd much rather write anything more than a handful of lines in a general purpose programmin…

> Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives. I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?" Statefu…

> tr '[A-Z]' '[a-z]' lower.txt

That's the biggest problem: some things are very simple, but other things fall off a cliff. For example, as a related task I ran into recently: how do you replace FOO with the contents of foo.txt? The natural way would be expanding it into a command line, but at least with sed that's no good even for nice short text files because / and \n are special. You can use a sed command to read a file which I didn't know existed until I looked it up, but it apparently has the delightful feature that "If file cannot be read for any reason, it is silently ignored and no error condition is set." You can use perl... you can use perl to easily do a lot of things that are really hard to do otherwise (including things as simple as matching a regex and printing capture groups), but at least to me it feels really awkward to wrong to mix two different full-fledged languages. Maybe I should just get over that, but I wish the whole thing were more coherent.

Re: Bash Pitfalls

#35

Post by jamesbritt »

wyclif wrote: post #11
jamesbritt wrote: post #9

Earlier quoted context omitted.

How would one know this in advance?

How would one know in advance that submitting to HN would result in traffic consisting of more than a handful of users?

How would one know in advance whether a site can handle high traffic?

And did I really have to spell this out?

Re: Bash Pitfalls

#36

Post by jfb »

comex wrote: post #7

The Unix shell may be a highly powerful interactive programming environment, but it's sure hard to think of anything that comes anywhere close to sucking as badly. With the shell and the standard Unix commands, some things that are hard in other languages are easy, and most of the things that are easy in other languages are hard to impossible... I'd love to see a clean slate replacement for the shell that still feels…

I think the problem is that the terrible shell semantics are inextricable from the larger semantics of using Unix. So you could replace sh with something less dumb-stupid, but you'd still be interacting with the garbage that is the shell environment. You could then replace the latter, but at that point, well, you're off into the weeds.

I hold my nose and write shell, even as I look over at, for instance, scsh, and think ... yeah.

Re: Bash Pitfalls

#37

Post by e12e »

comex wrote: post #34
sigil wrote: post #32

Earlier quoted context omitted.

> Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives. I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?" Statefu…

> tr '[A-Z]' '[a-z]' lower.txt That's the biggest problem: some things are very simple, but other things fall off a cliff. For example, as a related task I ran into recently: how do you replace FOO with the contents of foo.txt? The natural way would be expanding it into a command line, but at least with sed that's no good even for nice short text files because / and \n are special. You can use a sed command to read a…

Interesting problem. Some quick head-scratching and googling didn't turn up anything useful on merging templates with awk and sed... then it hit me --- m4 is used for that:

   sed -r 's/FOO/include(foo.txt)/g' temp.txt |m4

Re: Bash Pitfalls

#38

Post by derefr »

anon4 wrote: post #33

I know someone will sooner or later propose that we ban spaces and special characters in names. Let me just put my two cents forward. We should absolutely ban special characters from names. Specifically, all whitespace, the colon, semicolon, forward slash, backward slash, question mark, star, ampersand, and whatever else I'm missing that will confuse the shell. Also files cannot start with a dash. However, people sho…

> Substitute each problematic byte with equivalent percent-encoded form. This does not touch bytes over 0x80 - they are assumed non-problematic.

You know what's crazy? Currently, in Unix, control characters are allowed in filenames. Like, \t and \n and \b and even \[. Those shouldn't be allowed, percent-escaped or not. Everything else you said is sensible.

Re: Bash Pitfalls

#39

Post by comex »

e12e wrote: post #37
comex wrote: post #34

Earlier quoted context omitted.

> tr '[A-Z]' '[a-z]' lower.txt That's the biggest problem: some things are very simple, but other things fall off a cliff. For example, as a related task I ran into recently: how do you replace FOO with the contents of foo.txt? The natural way would be expanding it into a command line, but at least with sed that's no good even for nice short text files because / and \n are special. You can use a sed command to read a…

Interesting problem. Some quick head-scratching and googling didn't turn up anything useful on merging templates with awk and sed... then it hit me --- m4 is used for that: sed -r 's/FOO/include(foo.txt)/g' temp.txt |m4

Interesting solution; I should learn to use m4 for various tasks. Probably would have already if I didn't have such a negative visceral reaction to autotools :)

Re: Bash Pitfalls

#40

Post by agumonkey »

sigil wrote: post #32
brennen wrote: post #26

Earlier quoted context omitted.

I live and die by the shell. I'm constantly composing little one-liners, and keep an absurdly long Bash / zsh history to draw from. There are places the obvious answer is almost always "how about you just write a shell script?" That said, I long ago reached a place where I realized that, while shell scripting is entertaining , I'd much rather write anything more than a handful of lines in a general purpose programmin…

> Every single one of these questions is easier to answer if you're using a less agonizing language than Bash and its relatives. I disagree. While the set of things that are "hard" to do is probably larger in shell than the alternatives, the specific questions posed by the grandparent are hard in any language. They all boil down to "how can I correctly do something which has side effects (on external state)?" Statefu…

The polyglot is not a quality to me. Constantly serializing/parsing streams of text is pretty unpretty IMHO.
Post reply on HN