Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote an bash enumerator because I was sick of xargs

numerlab.org

Post reply on HN
Print view View original

51–60 of 202 posts

Re: I wrote an bash enumerator because I was sick of xargs

#51

Post by fn-mote »

dataflow wrote: post #14
tester457 wrote: post #9

On the topic of xargs replacements, I love gnu parallel. The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs. Parallel has an option for almost everything, it's almost too much. But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0] The gnu parallel book and reading mate…

Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning. Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way. Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted anot…

> Today I foolishly thought maybe I was the one who was holding it wrong […]

Apparently goes on to describe being confused about parallel reading from the standard input?

Re: I wrote an bash enumerator because I was sick of xargs

#52

Post by xorcist »

em-bee wrote: post #22
gumby wrote: post #20

Earlier quoted context omitted.

Because xargs is faster. Exec will invoke the command once per matching file (which is sometimes what you want, of course)! While xargs will accumulate a bunch of file names, then when it as n names will invoke the command with those names, while continuing to accumulate names until n is reached or the pipe closes. The size of n depends on the system, but is usually at least a thousand.

the example as given does not accumulate, so that is what i worked with. find -exec command '{}' '+' accumulates file arguments too. the only advantage of xargs is that you can tell it how many arguments to accumulate,̶ ̶t̶h̶e̶ ̶d̶o̶w̶n̶s̶i̶d̶e̶ ̶o̶f̶ ̶x̶a̶r̶g̶s̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶y̶o̶u̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶s̶p̶e̶c̶i̶f̶y̶ ̶t̶h̶e̶ ̶n̶u̶m̶b̶e̶r̶,̶ ̶w̶h̶e̶r̶e̶a̶s̶ ̶f̶i̶n̶d̶ ̶j̶u̶s̶t̶ ̶f̶i̶t̶s̶ ̶a̶s̶ ̶m̶a̶n̶y̶ ̶a̶s̶ ̶i̶t̶ ̶c…

The problem with find is that you can't specify the maximum tolerated command line lengths, and find hasn't historically been very smart about it (it just had a compiled-in value). Apparently this is fixed in GNU find, but for older systems and other platforms that may still be an issue.

Another feature missing in find that xargs has is the maximum processes to start at a time. find will run the commands in sequence, but in many situations you really want to run a bunch in parallel.

Re: I wrote an bash enumerator because I was sick of xargs

#53

Post by Walf »

chasil wrote: post #30

I use: find . -name '*.log' -print0 | xargs -0 rm For this simple example (derived from the article), find also has a delete operator. This null termination is now a POSIX standard.

Yeah, and I invariably add -r to xargs, so as not to execute the command unless results come through the pipeline.

If one is working with whole lines of text, setting the delimiter to newline is often desirable:

xargs -d \\n

Re: I wrote an bash enumerator because I was sick of xargs

#54

Post by em-bee »

xorcist wrote: post #52
em-bee wrote: post #22

Earlier quoted context omitted.

the example as given does not accumulate, so that is what i worked with. find -exec command '{}' '+' accumulates file arguments too. the only advantage of xargs is that you can tell it how many arguments to accumulate,̶ ̶t̶h̶e̶ ̶d̶o̶w̶n̶s̶i̶d̶e̶ ̶o̶f̶ ̶x̶a̶r̶g̶s̶ ̶i̶s̶ ̶t̶h̶a̶t̶ ̶y̶o̶u̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶s̶p̶e̶c̶i̶f̶y̶ ̶t̶h̶e̶ ̶n̶u̶m̶b̶e̶r̶,̶ ̶w̶h̶e̶r̶e̶a̶s̶ ̶f̶i̶n̶d̶ ̶j̶u̶s̶t̶ ̶f̶i̶t̶s̶ ̶a̶s̶ ̶m̶a̶n̶y̶ ̶a̶s̶ ̶i̶t̶ ̶c…

The problem with find is that you can't specify the maximum tolerated command line lengths, and find hasn't historically been very smart about it (it just had a compiled-in value). Apparently this is fixed in GNU find, but for older systems and other platforms that may still be an issue. Another feature missing in find that xargs has is the maximum processes to start at a time. find will run the commands in sequence,…

good points that i wasn't aware of, thank you. though personally i rarely start huge operations that need to be parallelized so i prefer simplicity over speed.

Re: I wrote an bash enumerator because I was sick of xargs

#55

Post by dylan604 »

xorcist wrote: post #49
eqvinox wrote: post #7

Not sure I see the point of this. Remembering a bunch of options on one tool is no better than a bunch of tools/constructs? Especially if the latter are useful elsewhere. And it can't be used for scripts unless you start shipping it alongside, which… nah. Just go with foo | while read X; do bar "$X"; done

Bash while loops are pretty readable and the above would be a nice to iterate over lines if it wasn't for that gnarly pipe, which is a common source of errors in this construct. Remember that a pipe starts a new shell. So: grep stuff file.txt | while read key value ; do [ "$key" = "target" ] && found="$value" ; done where you might expect $found to end up with the value for the line that has "target" in the first col…

> and isn't compatible

is that a GNU v POSIX type of compatibility issue?

Re: I wrote an bash enumerator because I was sick of xargs

#57

Post by samtheprogram »

laughing_man wrote: post #50
samtheprogram wrote: post #17

Literally just use xargs with -I {} and quotation marks?

That won't save you from weird file names, but null termination will.

Passing "{}" handles most sane cases including spaces. If I'm doing bash that needs to be robust (rare and/or dotfiles) or know the folder/dataset has weird filenames, sure.

Re: I wrote an bash enumerator because I was sick of xargs

#58

Post by teo_zero »

Good idea but strange syntax. It would be more idiomatic if the command came first and the list of globs last.

Additionally the use of "--" is not what everybody expects: here it is used to introduce one argument, the command, while it's usually meant to introduce multiple arguments without worrying if they have a leading "-".

A possible revised syntax with command as the first argument followed by a list of globs optionally introduced by "--" would allow to enumerate all files with a leading "-", which the current syntax cannot:

  enumerate 'whatever {}' -- '-*'
I'm assuming "-f" for simplicity, but the same reasoning holds for "-L" too.

Re: I wrote an bash enumerator because I was sick of xargs

#59

Post by sidewndr46 »

tester457 wrote: post #9

On the topic of xargs replacements, I love gnu parallel. The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs. Parallel has an option for almost everything, it's almost too much. But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0] The gnu parallel book and reading mate…

isn't parallel that tool that dumps some kind of begging message into your output each time you invoke it?

Re: I wrote an bash enumerator because I was sick of xargs

#60

Post by rmunn »

ivlad wrote: post #34
chasil wrote: post #30

I use: find . -name '*.log' -print0 | xargs -0 rm For this simple example (derived from the article), find also has a delete operator. This null termination is now a POSIX standard.

Was exactly my thought: why not to use null termination? Looks like a case where reading man page would have spared writing another copycat utility.

And null termination is guaranteed to work, because the only two characters forbidden in Unix filenames (for most varieties of Unix, I won't guarantee there aren't some weird variants out there) are / and null.

The only times I've needed something more than `find -print0 | xargs -0` has been when I need to apply logic to decide whether to process one of the files, in a way that's not easy to express in a `find` command. Then I write a small script with a for loop and if statements inside it.

But more people should know about `-print0`. It's the answer to 95% of the problems with `find | xargs`.

Post reply on HN