Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A common mistake involving wildcards and the find command

blog.robertelder.org

Post reply on HN
Print view View original

31–40 of 148 posts

Re: A common mistake involving wildcards and the find command

#31

Post by tyingq »

seiferteric wrote: post #9

This was obvious to me, but one version of this that surprised me is when using scp. If you glob a remote destination like "scp myserver:*.jpg ./" It will probably work! But how? Because the remote path will likely not match any local files and the path with the asterisk will be passed to scp and scp will do the globbing on the remote side.

I mostly use ssh and tar instead of scp, because I'm usually after more than one file.

Like, "ssh remote 'cd /whatever; tar -cf - someglobpattern' | tar -xvf"

Re: A common mistake involving wildcards and the find command

#32

Post by tzs »

Another one I've seen a few times with find (although it is actually more an xargs thing than a find thing), usually not with any bad consequences at least, is something like this:

find . -type f | xargs grep foo

You expect to see all the "foo" lines from your files, each prefixed with the file name and a colon. And that's what you get most of the time.

But sometimes you might get a foo line without the filename prefix.

Why? Because grep only adds the filename prefix when there is more than one filename argument. There are two ways that might come about in the above command.

The first is if the find only finds one file.

The second is if the find finds so many files that xargs has to invoke grep more than once. It can happen that there is only one file left to do when it gets to the final grep invocation.

Simple fix:

find . -type f | xargs grep foo /dev/null

Re: A common mistake involving wildcards and the find command

#33

Post by shmageggy »

joana035 wrote: post #28

The title seems a bit off since shell expansions and arguments has nothing to do with the find command. Both features are also often covered in entry level material for introduction to shell.

Well it seems the root of the problem is

> * Most importantly, the 'find' command uses a different algorithm than shell globbing does when matching wildcard characters. More specifically, the find command will apply the search pattern against the base of the file name with all leading directories removed. This is contrasted from shell globbing which will expand the wildcard between each path component separately. When no path components are specified, the wildcard will match only files in the current directory.*

So there does seem to be a `find` specific issue here

Re: A common mistake involving wildcards and the find command

#34

Post by kalium_xyz »

tzs wrote: post #26

For a long time (probably since the first time I forgot to quote something and got burned, so around 40 years), I've thought that there should be some mechanism for the shell to pass in information about how each argument came about. For each argument, it would tell the program if it was supplied directly, or came from wildcard expansion. For those from wildcard expansion, it would tell the program what the wildcard…

The unix haters guide lists the shell handling the wildcard like it does as a major flaw with unix shells.

Re: A common mistake involving wildcards and the find command

#35

Post by nothrabannosir »

johncs wrote: post #23
twic wrote: post #10

I often set the nullglob option in scripts, because it makes the handling of globs which don't match anything a bit more predictable: http://bash.cumulonim.biz/NullGlob.html There's a note at the end about how with nullglob set, ls on a glob with no matches does something surprising. This is a great illustration of how an empty list and the absence of a list are different. Sadly it's rather hard to make that distinct…

I set failglob: `shopt -s failglob`. Makes the whole command fail if there's no matches. That combined with `set -e` which aborts the script in the event of any command failing makes me feel somewhat safe. Indeed I add the following two lines to every bash script I write: set -exu shopt -s failglob

If you like set -e, I recommend looking at set -o pipefail.

Re: A common mistake involving wildcards and the find command

#38

Post by nothrabannosir »

Tangential: another safeguard you can adopt is avoiding "hard delete" commands like rm and find -delete. Untrain yourself from these commands by never using them. On Mac systems, the "trash" program (brew install trash) sends files to your system trash. You can use `trash [file]` and `find .. -print0 | xargs -0 trash --`. rm is a dangerous command you should only very rarely be using.

I fish something out of the trash a few times a year and lemme tell ya; it's worth the investment.

Another tip if you fancy debugging shells is using

  python -c "print(__import__('sys').argv[1:])" sample "arg here" * foo
this provides the same functionality as the c program in TFA, without needing gcc.

Re: A common mistake involving wildcards and the find command

#39

Post by Xophmeister »

tzs wrote: post #32

Another one I've seen a few times with find (although it is actually more an xargs thing than a find thing), usually not with any bad consequences at least, is something like this: find . -type f | xargs grep foo You expect to see all the "foo" lines from your files, each prefixed with the file name and a colon. And that's what you get most of the time. But sometimes you might get a foo line without the filename pref…

A “better” fix would be:

    find . -type f -exec grep -H foo {} +

Re: A common mistake involving wildcards and the find command

#40

Post by simias »

twic wrote: post #10

I often set the nullglob option in scripts, because it makes the handling of globs which don't match anything a bit more predictable: http://bash.cumulonim.biz/NullGlob.html There's a note at the end about how with nullglob set, ls on a glob with no matches does something surprising. This is a great illustration of how an empty list and the absence of a list are different. Sadly it's rather hard to make that distinct…

I don't know if it's the default or if it's something I've set in my config a decade or two ago but my zsh behaves like this by default (i.e. I get an error for blobs that match nothing instead of silently passing the * along). That seems much saner to me:

    $ find . -name *.txt
    zsh: no matches found: *.txt
That'll teach you to quote your `find` patterns real fast...
Post reply on HN