Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

111–120 of 135 posts

Re: An Opinionated Guide to Xargs

#111

Post by chubot »

rcpt wrote: post #107
chubot wrote: post #31

Earlier quoted context omitted.

This is like the sed|bash anti-pattern mentioned in the original post, and quoted in the appendix on shell injection. I wouldn't say "never use it", but I would hesitate to ever put it in a script, vs. doing a one-off at the command line.

You don't pipe to bash on the first run. Use awk/sed without piping to workshop your commands. Once you've got them right send over to bash. This is far superior to futzing with xargs interactive or whatever dry run feature they have.

See the string injection comment:

http://www.oilshell.org/blog/2021/08/xargs.html#more-comment...

https://lobste.rs/s/wlqveb/xargs_considered_harmful#c_kwsxtc

Re: An Opinionated Guide to Xargs

#112

Post by thayne »

I'm not sure I like the `$1` and shell function pattern. It might avoid the -I minilanguage, but at the cost of "being clever" in a way that takes a minute to wrap your head around. It's a neat trick, but I don't think it would be easy to understand if you are reading the code for the first time.

Re: An Opinionated Guide to Xargs

#113

Post by MainJane »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

It is documented in the GNU Parallel documentation: https://www.gnu.org/software/parallel/parallel_alternatives....

If you seriously believe you can implement everything using xargs, then this (contrived) example is for you: https://unix.stackexchange.com/questions/405552/using-xargs-...

Newer versions include 'parset' which can set shell variables in parallel, which is useful if you want to 'map' values from one array to another.

Re: An Opinionated Guide to Xargs

#114

Post by MainJane »

orhmeh09 wrote: post #96
queuebert wrote: post #86

Earlier quoted context omitted.

It's also written in Perl!

Veering off course here, after experiencing how incredibly long it took to install Sqitch, I will go out of my way to avoid anything that is more than a single script, certainly anything requiring CPAN too. I don’t think there’s anything technically wrong with these programs or with Perl, they’re just presented in ways that are unique hassles in this day and age.

> anything that is more than a single script

... which is exactly what GNU Parallel is. Your concern is even mentioned in the design documentation: https://www.gnu.org/software/parallel/parallel_design.html

Re: An Opinionated Guide to Xargs

#115

Post by queuebert »

orhmeh09 wrote: post #96
queuebert wrote: post #86

Earlier quoted context omitted.

It's also written in Perl!

Veering off course here, after experiencing how incredibly long it took to install Sqitch, I will go out of my way to avoid anything that is more than a single script, certainly anything requiring CPAN too. I don’t think there’s anything technically wrong with these programs or with Perl, they’re just presented in ways that are unique hassles in this day and age.

I remember when Perl was the coolest thing ever. I even wrote numerical simulations in it, just to try. Only with the invention of Python and Ruby did we realize how much better things could be. Of course that the Perl inventor was an IOCCC winner should've been a red flag.

Re: An Opinionated Guide to Xargs

#116

Post by MainJane »

chubot wrote: post #59
comex wrote: post #27

Earlier quoted context omitted.

One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at…

OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. Y…

Your `do_one`:

  * does not buffer stderr
  * does not check if the disk is full for a period of time during a task (thus risking incomplete output)
  * does not clean up, if killed
  * does not work correctly if task_with_stdout is a composed command
Given that GNU Parallel is a drop-in replacement for xargs, I am curious why you find it a 'weirdo interface'.

Re: An Opinionated Guide to Xargs

#117

Post by MainJane »

orhmeh09 wrote: post #68
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

The nagware prompts of parallel are so objectionable that I will do a lot of things to avoid using it at all. So pretentious!

Have you read the FAQ?

https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...

Re: An Opinionated Guide to Xargs

#118

Post by O_H_E »

vhold wrote: post #28
orf wrote: post #20

Earlier quoted context omitted.

Resumption, error reporting and much better progress monitoring.

Oh I didn't know about resumption.. parallel has so many features packed into its CLI it's kind of ridiculous. For others that didn't know about it, see the examples here: https://www.gnu.org/software/parallel/parallel_tutorial.html... Here's another surprising feature: https://www.gnu.org/software/parallel/parallel_tutorial.html...

Wow!, That is surprising and potentially very useful.

Re: An Opinionated Guide to Xargs

#120

Post by cormacrelf »

WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

If you need more visibility into long running processes, pueue is another alternative. You can of course use `xargs -P1 pueue add ./process_file.sh` to add the jobs in the first place. Sends a job to pueued, returns immediately. Great for re-encoding dozens of videos. For jobs that aren’t already multi-core, set the queue parallelism with pueue, after you’ve seen your cpu is under-utilised.

Obviously downside to the visibility and dynamism is that it redirects stdout. You can read it back later, in order. But it’s not there for continued processing immediately.

Post reply on HN