Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

61–70 of 135 posts

Re: An Opinionated Guide to Xargs

#61

Post by chubot »

jeffbee wrote: post #41
chubot wrote: post #11

Earlier quoted context omitted.

(author here) Hm I don't see either of these points because: GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post) xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds. -- edit: Logging can also be done with by recursively invoking sh…

Parallel's killer feature is how it spools subprocess output, ensuring that it doesn't get jumbled together. xargs can't do that. I use parallel for things like shelling out to 10000 hosts and getting some statistics. If I use xargs the output stomps all over itself.

Ah OK thanks, I responded to this here: https://news.ycombinator.com/item?id=28259473

Re: An Opinionated Guide to Xargs

#62

Post by fiddlerwoaroof »

aaaaaaaaaaab wrote: post #58
fiddlerwoaroof wrote: post #55

Earlier quoted context omitted.

Yeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too. I think the redirection can come first, though (not at a computer to test):

Yeah, for commands , the input/output redirections can precede them, but for some reason it doesn't work for builtin constructs like `while`: $

Maybe wrap the loop either with parentheses or braces?

Re: An Opinionated Guide to Xargs

#63

Post by blibble »

jeffbee wrote: post #40
LeoPanthera wrote: post #36

Earlier quoted context omitted.

Yeah but xargs doesn't refuse to run until I have agreed to a EULA stating I will cite it in my next academic paper.

parallel doesn't either, it just nags. I agree about how silly and annoying it is. Imagine if every time the parallel author opened Firefox he got a message reminding him to personally thank me if he uses his web browser for research, or if every time his research program calls malloc he has to acknowledge and cite Ulrich Drepper. Very very silly. Parallel is the better tool but the nagware impairs its reputation.

or every time a process called fork() you had to read some stupid message

Re: An Opinionated Guide to Xargs

#64

Post by figomore »

xmcqdpt2 wrote: post #53
chubot wrote: post #30

Earlier quoted context omitted.

I'd like to see a demo of it! I will try rewriting it with the $0 Dispatch Pattern and ssh :)

Good luck balancing node usage! Here is an example of how it works, https://docs.computecanada.ca/mediawiki/index.php?title=GNU_... This + restart capabilities make gnu parallel very well suited to running 1000s of compute-heavy jobs on HPC clusters.

I used Parallel to distribute the rendering of a little Blender animation It worked very well.

https://github.com/tfmoraes/blender_gnu_parallel_render/blob...

Re: An Opinionated Guide to Xargs

#65

Post by aaaaaaaaaaab »

fiddlerwoaroof wrote: post #62
aaaaaaaaaaab wrote: post #58

Earlier quoted context omitted.

Yeah, for commands , the input/output redirections can precede them, but for some reason it doesn't work for builtin constructs like `while`: $

Maybe wrap the loop either with parentheses or braces?

Tried that, but nope :D I'll let you figure this one out once you get near a computer!

Re: An Opinionated Guide to Xargs

#66

Post by chriswarbo »

aaaaaaaaaaab wrote: post #47
fiddlerwoaroof wrote: post #22

I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

Some additional tips: 1. You don't need the parentheses. 2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope: i=0 while read -r v; do ... i=$(( i + 1)) done The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯ [1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

I use this exact pattern a lot. One thing to consider is that in the process substitution version, do_something can't modify the enclosing variables. The vast majority of the time I want to modify variables in the loop body and not the generating process, but it's worth keeping in mind.

One common pattern I use this for is running a bunch of checks/tests, e.g.

    EXIT_CODE=0
    while read -r F
    do
        do_check "$F" || EXIT_CODE=1
    done
This is a more complicated alternative to the following:

    find ./tests -type f | while read -r F
    do
      do_check "$F" || exit 1
    done
The simpler version will abort on the first error, whilst the first version will always run all of the checks (exiting with an error afterwards, if any of them failed)

Re: An Opinionated Guide to Xargs

#67

Post by Godel_unicode »

chubot wrote: post #59
comex wrote: post #27

Earlier quoted context omitted.

One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at…

OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. Y…

Thank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python.

For what it's worth, I consider oil to be closer to a unixy PowerShell rather than a more powerful bash. Note that this is not a slight, PowerShell is sweet for what it is. It (oil) really takes a hard left from the POSIX philosophy of focusing on one thing and doing it well. I'm also bitter that, if it's going to veer so far away from POSIX, that it didn't go the whole hundred and become a function language with comprehensions and such.

For what it's worth, everything you mentioned above about your approach can be done with parallel.

Re: An Opinionated Guide to Xargs

#68

Post by orhmeh09 »

WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

The nagware prompts of parallel are so objectionable that I will do a lot of things to avoid using it at all. So pretentious!

Re: An Opinionated Guide to Xargs

#69

Post by orhmeh09 »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

Issue complaint prompts to promote the author, for one.
Post reply on HN