Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

51–60 of 135 posts

Re: An Opinionated Guide to Xargs

#51

Post by bloopernova »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

GNU Parallel can be sourced into a bash session from a plain text file and used as a function. I've used it to get around overly-restrictive build environments. (overly restrictive because the team that manages the build image wasn't open to modifying their image for my use case)

Re: An Opinionated Guide to Xargs

#52

Post by aaaaaaaaaaab »

fiddlerwoaroof wrote: post #22

I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

Also for the `while` enthusiasts, here's how you zip the output of two processes in bash:

    paste -d \\n

Re: An Opinionated Guide to Xargs

#53

Post by xmcqdpt2 »

chubot wrote: post #30
leephillips wrote: post #24

Earlier quoted context omitted.

Remote execution.

I'd like to see a demo of it! I will try rewriting it with the $0 Dispatch Pattern and ssh :)

Good luck balancing node usage!

Here is an example of how it works,

https://docs.computecanada.ca/mediawiki/index.php?title=GNU_...

This + restart capabilities make gnu parallel very well suited to running 1000s of compute-heavy jobs on HPC clusters.

Re: An Opinionated Guide to Xargs

#54

Post by xmcqdpt2 »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

Restart capability and remote executions make gnu parallel the tool if choice for HPC. For example, you might very well use gnu parallel to run 1000s of cpu-hours of numerical simulation using patterns such as these ones,

https://docs.computecanada.ca/mediawiki/index.php?title=GNU_...

Using xargs for this kind of work is euhm... not a good idea.

Re: An Opinionated Guide to Xargs

#55

Post by fiddlerwoaroof »

aaaaaaaaaaab wrote: post #47
fiddlerwoaroof wrote: post #22

I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

Some additional tips: 1. You don't need the parentheses. 2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope: i=0 while read -r v; do ... i=$(( i + 1)) done The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯ [1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

Yeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too.

I think the redirection can come first, though (not at a computer to test):

Re: An Opinionated Guide to Xargs

#56

Post by bialpio »

bialpio wrote: post #50
jmholla wrote: post #25

Earlier quoted context omitted.

Can you expand on that? I've never had trouble leveraging xargs and find it aligns well with shell piping.

Not OP but to me the best thing about PowerShell is that it recognizes that text is not always the best way to output results from commands if you care about creating pipelines. In short, it passes objects around so there's no need for parsing text.

Two examples from the article translated into PS (sorry, I'm a bit rusty so the second one may not be the shortest possible):

  PS> "alice", "bob" | echo

  PS> Get-ChildItem . -Include "*test.cpp","*test.py" -Recurse | foreach { Remove-Item $_.Name }
No text parsing in sight, and the object attributes can be tab-completed from the shell (e.g. I tab-completed the `$_.Name`).

Re: An Opinionated Guide to Xargs

#57

Post by derriz »

aaaaaaaaaaab wrote: post #48

I would recommend using -0 instead of -d, as the latter is not supported on BSD (and macOS) xargs: do_something | tr \\n \\0 | xargs -0 ...

I wish this was the default behavior of xargs (the 'tr \\n \\0 | xargs -0' bit). I don't know why xargs splits on spaces and tabs as well as newlines by default and doesn't even have a flag to just split on lines.

Ok filenames can theoretically have newlines in them but I'd be happy to deal with that weird case. I can't recall ever having encountered it in years of using bash on various systems.

Shell pipes would then orthogonally provide the stuff like substitution that xargs does in it's own unique way (that I just can't be bothered learning) - instead you'd just pipe the find output through sed or 'grep -v' or whatever you wanted before piping into xargs.

I guess that's what aliases but I'm too lazy anymore to bother with configuring often short-lived systems all the time.

Re: An Opinionated Guide to Xargs

#58

Post by aaaaaaaaaaab »

fiddlerwoaroof wrote: post #55
aaaaaaaaaaab wrote: post #47

Earlier quoted context omitted.

Some additional tips: 1. You don't need the parentheses. 2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope: i=0 while read -r v; do ... i=$(( i + 1)) done The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯ [1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

Yeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too. I think the redirection can come first, though (not at a computer to test):

Yeah, for commands, the input/output redirections can precede them, but for some reason it doesn't work for builtin constructs like `while`:

    $

Re: An Opinionated Guide to Xargs

#59

Post by chubot »

comex wrote: post #27
chubot wrote: post #16

Earlier quoted context omitted.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at…

OK thanks, looks like there are several features of GNU parallel that users like.

For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file:

    do_one() {
      task_with_stdout > $dir/$task_id.txt
    }
So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them.

You even have some notion of progress by checking the file size with ls -l.

I tend to use a pattern where each task also outputs a metadata file: the exit status, along with the data from "time" (rusage, etc.)

But I admit that this is annoying to rewrite in every script that uses xargs! It does make sense to have this functionality in a tool.

But I think that tool should be a LANGUAGE like Oil, not a weirdo interface like GNU parallel :)

But thanks for the explanation (and thanks to everyone in this subthread) -- I learned a bunch and this is why I write blog posts :)

Re: An Opinionated Guide to Xargs

#60

Post by entire-name »

fiddlerwoaroof wrote: post #55
aaaaaaaaaaab wrote: post #47

Earlier quoted context omitted.

Some additional tips: 1. You don't need the parentheses. 2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope: i=0 while read -r v; do ... i=$(( i + 1)) done The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯ [1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

Yeah, although I use the parentheses mostly because I like how it reads. And that process substitution trick is important too. I think the redirection can come first, though (not at a computer to test):

Redirection like this doesn't seem to work if it comes first on GNU bash 5.0.17(1)-release.

For documentation purposes, this is the exact thing I tried to run:

    $
Maybe there is another way...
Post reply on HN