Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A common mistake involving wildcards and the find command

blog.robertelder.org

Post reply on HN
Print view View original

51–60 of 148 posts

Re: A common mistake involving wildcards and the find command

#51

Post by jolmg »

tzs wrote: post #26

For a long time (probably since the first time I forgot to quote something and got burned, so around 40 years), I've thought that there should be some mechanism for the shell to pass in information about how each argument came about. For each argument, it would tell the program if it was supplied directly, or came from wildcard expansion. For those from wildcard expansion, it would tell the program what the wildcard…

> For those from wildcard expansion, it would tell the program what the wildcard was.

Different shells have different globbing mechanisms. Why should all programs tie themselves to the mechanisms of any one particular shell?

The simpler the calling mechanism for executables, the simpler it is to write them in any existing or future language. This also gives more flexibility to future shells.

UNIX is pretty much designed thinking of users as programmers. Making it easy to write programs building on other programs is as important as being able to call them. With that in mind, I don't think it's a good compromise to complicate the writing of executables in order to protect users from their own mistakes.

Re: A common mistake involving wildcards and the find command

#52

Post by Cpoll »

nothrabannosir wrote: post #38

Tangential: another safeguard you can adopt is avoiding "hard delete" commands like rm and find -delete. Untrain yourself from these commands by never using them. On Mac systems, the "trash" program (brew install trash) sends files to your system trash. You can use `trash [file]` and `find .. -print0 | xargs -0 trash --`. rm is a dangerous command you should only very rarely be using. I fish something out of the tras…

I learned from a sysadmin to use `mv` instead of `rm` when you're removing potentially critical files.

Other tips include colouring your prompt highlighted red on production boxes so you never accidentally think you're somewhere safe.

Nowadays I try to avoid SSHing into mission critical machines though.

Re: A common mistake involving wildcards and the find command

#53

Post by cellularmitosis »

dmurray wrote: post #45
Xophmeister wrote: post #39

Earlier quoted context omitted.

A “better” fix would be: find . -type f -exec grep -H foo {} +

Yeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.

Why is find | grep an anti-pattern? Is eschewing memorizing the flags of one particular command in favor of using a pipeline which is much more broadly applicable really an anti-pattern?

Heck, I’m at the point now where I pipe tar into gzip.

Re: A common mistake involving wildcards and the find command

#54

Post by nine_k »

tyingq wrote: post #31
seiferteric wrote: post #9

This was obvious to me, but one version of this that surprised me is when using scp. If you glob a remote destination like "scp myserver:*.jpg ./" It will probably work! But how? Because the remote path will likely not match any local files and the path with the asterisk will be passed to scp and scp will do the globbing on the remote side.

I mostly use ssh and tar instead of scp, because I'm usually after more than one file. Like, "ssh remote 'cd /whatever; tar -cf - someglobpattern' | tar -xvf"

Could also be faster because of the common compression dictionary.

Re: A common mistake involving wildcards and the find command

#55

Post by detaro »

derekp7 wrote: post #50
dmurray wrote: post #45

Earlier quoted context omitted.

Yeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.

xargs is also more efficient than -exec, as it stuffs multiple files to the target command. Esp. for rm, instead of executing a copy for each file.

doesn't

    -exec  {} \+
do that too? (and we're back at the "remembering specific syntax" point: knowing xargs helps you everywhere)

Re: A common mistake involving wildcards and the find command

#56

Post by Xophmeister »

derekp7 wrote: post #50
dmurray wrote: post #45

Earlier quoted context omitted.

Yeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.

xargs is also more efficient than -exec, as it stuffs multiple files to the target command. Esp. for rm, instead of executing a copy for each file.

The “+” terminator to -exec has the same effect for find.

Re: A common mistake involving wildcards and the find command

#57

Post by anonsivalley652 »

svnpenn wrote: post #16

As another user mentioned, many POSIX and/or GNU utilities havent aged well. I respect trying to stay portable, but peoples needs change over time and these tools simply havent kept up. Like the other user, I use Fd now instead: https://github.com/sharkdp/fd As well as Silver Searcher: https://github.com/ggreer/the_silver_searcher while Grep performance is pretty good, its also gotten pretty stale with regard to its…

For grepping, rg is faster and saner.

https://github.com/BurntSushi/ripgrep

On end-user systems with fast I/O, it is usually a better use of resources to have a battery-/suspend-/reboot-aware indexing system that has path/extension white/blacklists and prioritizes file monitored changes. This way, searching can happen against an optimized text search DBMS much faster than waiting for zillions of IOPS that are basically wasted by instead spending them ahead-of-time in the background/idle indexing text files/metadata/structured data once when it changed/s and already know exactly where all occurrences of kWhateverCondition_PP3V42_G3H or \A[AB]{1,3}c+d\z live under ~/Projects without reading any actual files.

Re: A common mistake involving wildcards and the find command

#58

Post by Xophmeister »

cellularmitosis wrote: post #53
dmurray wrote: post #45

Earlier quoted context omitted.

Yeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.

Why is find | grep an anti-pattern? Is eschewing memorizing the flags of one particular command in favor of using a pipeline which is much more broadly applicable really an anti-pattern? Heck, I’m at the point now where I pipe tar into gzip.

I assume you mean “find | xargs”, rather than “find | grep”; the latter would grep against the paths that find matches, rather than their contents. Anyway, the reason “find | xargs” is considered an antipattern is because xargs splits its input on newlines and this would break on any paths that contained a newline. Arguably, you can get around this by using find’s -print0 and xargs’s -0 arguments, respectively, as paths cannot contain null bytes, but you might as well have used -exec at this point!

Re: A common mistake involving wildcards and the find command

#59

Post by gdavisson »

emmelaich wrote: post #47
tyingq wrote: post #31

Earlier quoted context omitted.

I mostly use ssh and tar instead of scp, because I'm usually after more than one file. Like, "ssh remote 'cd /whatever; tar -cf - someglobpattern' | tar -xvf"

Tip: get into the habit of using && rather than ; So... > ssh remote 'cd /whatever && tar -cf - someglobpattern' | tar -xvf

Seconded. Here https://apple.stackexchange.com/questions/378942/cant-initia... is an example of what can happen from not checking for errors after a `cd` command. Summary: "sudo find ... -delete" ran in the root directory rather than the intended location, and now the OS is gone (probably along with lots more).

Re: A common mistake involving wildcards and the find command

#60

Post by jrockway »

cellularmitosis wrote: post #53
dmurray wrote: post #45

Earlier quoted context omitted.

Yeah, find ¦ xargs is definitely an antipattern, but in practice I use it all the time. Either because I forget the syntax for find -exec, or because I'm feeling principled that find -exec breaks the Unix philosophy of doing one thing well and being composable, or because my script started using ls and got changed to use find.

Why is find | grep an anti-pattern? Is eschewing memorizing the flags of one particular command in favor of using a pipeline which is much more broadly applicable really an anti-pattern? Heck, I’m at the point now where I pipe tar into gzip.

I think there is a cottage industry around telling people that they're using the interactive shell wrong. This all started with "useless use of cat", which people still bitch about on Reddit and Twitter on a regular basis, and so whenever someone finds something a little weird about UNIX, they are quick to call a person asking a question about it dumb. I think it might be stockholm syndrome, or a symptom of one's greatest achievement in life thusfar being reading the man page for find. Or maybe they genuinely think they're helping. I dunno.

Ultimately, it's a cascading failure of bad design. Good design is that find and xargs each do one thing and do it well. Find can output files to anything! Xargs can xargify input from anything! Bad design is that find uses \n as the output record separator and that xargs uses \n as the input record separator, but \n can appear in filenames! This design can never work and will always lead to difficult-to-debug problems. So instead of fixing it, we blame the user for not knowing "oh, well find has xargs built in. sort of. a lot of the features are there. not all of them. but some. so never use find | args, smile." or "well, we designed the unix shell to be easy to use... but there is this corner case that we knew about and could have protected you from, but we chose not to, so 1% of the time you'll rm -rf / with your xargs command, UNLESS YOU REMEMBER THIS ONE WEIRD TRICK which is to use -0 to use an out-of-band symbol to separate records which we could have just made happen by default but chose not to, just to mess you up!"

I appreciate the efforts of people that are re-imagining the interactive shell, like PowerShell. I am not very productive with PowerShell (to get a listing of files in your directory, just type "Could-We-Make-It-Even-More-Verbose -PerhapsInTheNextVersionWeWill!") but it's a good prototype of a good shell. I have personally stepped on enough UNIX landmines to be generally OK with the compromises, or more typically don't write shell scripts for anything that will be run more than twice. It's bad though, and I don't think the users are to blame.

Post reply on HN