Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

11–20 of 135 posts

Re: An Opinionated Guide to Xargs

#11

Post by chubot »

westurner wrote: post #9

Wanting verbose logging from xargs, years ago I wrote a script called `el` (edit lines) that basically does `xargs -0` with logging. https://github.com/westurner/dotfiles/blob/develop/scripts/e... It turns out that e.g. -print0 and -0 are the only safe way: line endings aren't escaped: find . -type f -print0 | el -0 --each -x echo GNU Parallel is a much better tool: https://en.wikipedia.org/wiki/GNU_parallel

(author here) Hm I don't see either of these points because:

GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post)

xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds.

--

edit: Logging can also be done with by recursively invoking shell functions that log with the $0 Dispatch Pattern, explained in the post. I don't see a need for another tool; this is the Unix philosophy and compositionality of shell at work :)

Re: An Opinionated Guide to Xargs

#13

Post by chubot »

MichaelGroves wrote: post #7

> A lobste.rs user asked why you would use find | xargs rather than find -exec. The answer is that it can be much faster. If you’re trying to rm 10,000 files, you can start one process instead of 10,000 processes! Fair enough, but I still favor find -exec . I find it generally less error prone, and it's never been so slow that I wished I had instead used xargs. Also, if you're specifically using -exec rm with find, y…

A benefit I didn't mention in the post (but probably should) is that the pipe lets you interpose other tools.

That is, find -exec is sort of "hard-coded", while find | xargs allows obvious extensions like:

    find | grep | xargs   # filter tasks

    find | head | xargs   # I use this all the time for faster testing

    find | shuf | xargs
Believe it or not I actually use find | shuf | xargs mplayer to randomize music and videos :)

So shell is basically a more compositional language than find (which is its own language, as I explain here: http://www.oilshell.org/blog/2021/04/find-test.html )

Re: An Opinionated Guide to Xargs

#14

Post by masklinn »

bobbylarrybobby wrote: post #8
MichaelGroves wrote: post #7

> A lobste.rs user asked why you would use find | xargs rather than find -exec. The answer is that it can be much faster. If you’re trying to rm 10,000 files, you can start one process instead of 10,000 processes! Fair enough, but I still favor find -exec . I find it generally less error prone, and it's never been so slow that I wished I had instead used xargs. Also, if you're specifically using -exec rm with find, y…

You can also use `find -exec` with `'+'` instead of `';'` as the terminator. This will call `rm` on all of the found files in one call.

I tend to prefer xargs because it works in more contexts e.g. I've got a tool which automatically generates databases but sometimes the cleanup doesn't work. `find -exec` does nothing, but `xargs -n1 dropdb` (following an intermediate grep) does the job. From there, it makes sense to… just use xargs everywhere.

And I always fail to remember that the -exec terminator must be escaped in zsh, so using -exec always takes me multiple tries. So I only use -exec when I must (for `find` predicates).

Re: An Opinionated Guide to Xargs

#16

Post by chubot »

WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut...

What does it do that xargs and shell can't? (honest question)

Re: An Opinionated Guide to Xargs

#17

Post by ineedasername »

Zababa wrote: post #5
legobmw99 wrote: post #2

This is only tangentially related, but after all the posts here the last few days about thought terminating cliches, I can’t help but reflect on the “X considered harmful” title cliche

I've been thinking about titles, and it's hard to make a good one that doesn't look like a total cliché. "X considered harmful", "an opinionated guide to X", some kind of joke or reference, what could be a collection of tags (X, Y and Z), "things I have learned doing X", etc.

In this case a less cliche/click-baity title could simply be:

"A Response to Xargs Criticism"

Re: An Opinionated Guide to Xargs

#18

Post by lacksconfidence »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

i don't know if xargs cant, but i use gnu parallel to split an input pipe into N parallel pipes processing slices of the input stream.

Edit: To clarify, xargs usually wants to spin up a process per task. I have parallel spin up N processes and then continuously feed them.

Re: An Opinionated Guide to Xargs

#19

Post by masklinn »

> Shell functions and $1, instead of xargs -I {}

> -n instead of -L (to avoid an ad hoc data language)

Apparently GNU xargs is missing it, but BSD xargs has -J, which is a `-I` which works with `-n`: with `-I` each replstr gets replaced by one of the inputs, with `-J` the replstr gets replaced by the entire batch (as determined by `-n`).

Re: An Opinionated Guide to Xargs

#20

Post by orf »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

Resumption, error reporting and much better progress monitoring.
Post reply on HN