Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

41–50 of 135 posts

Re: An Opinionated Guide to Xargs

#41

Post by jeffbee »

chubot wrote: post #11
westurner wrote: post #9

Wanting verbose logging from xargs, years ago I wrote a script called `el` (edit lines) that basically does `xargs -0` with logging. https://github.com/westurner/dotfiles/blob/develop/scripts/e... It turns out that e.g. -print0 and -0 are the only safe way: line endings aren't escaped: find . -type f -print0 | el -0 --each -x echo GNU Parallel is a much better tool: https://en.wikipedia.org/wiki/GNU_parallel

(author here) Hm I don't see either of these points because: GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post) xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds. -- edit: Logging can also be done with by recursively invoking sh…

Parallel's killer feature is how it spools subprocess output, ensuring that it doesn't get jumbled together. xargs can't do that. I use parallel for things like shelling out to 10000 hosts and getting some statistics. If I use xargs the output stomps all over itself.

Re: An Opinionated Guide to Xargs

#42

Post by Godel_unicode »

chubot wrote: post #11
westurner wrote: post #9

Wanting verbose logging from xargs, years ago I wrote a script called `el` (edit lines) that basically does `xargs -0` with logging. https://github.com/westurner/dotfiles/blob/develop/scripts/e... It turns out that e.g. -print0 and -0 are the only safe way: line endings aren't escaped: find . -type f -print0 | el -0 --each -x echo GNU Parallel is a much better tool: https://en.wikipedia.org/wiki/GNU_parallel

(author here) Hm I don't see either of these points because: GNU xargs has --verbose which logs every command. Does that not do what you want? (Maybe I should mention its existence in the post) xargs -P can do everything GNU parallel do, which I mention in the post. Any counterexamples? GNU parallel is a very ugly DSL IMO, and I don't see what it adds. -- edit: Logging can also be done with by recursively invoking sh…

As far as I'm aware, xargs still has the problem of multiple jobs being able to write to stdout at the same time, potentially causing their output streams to be intermingled. Compare this with parallels --group.

Also parallels can run some of those threads on remote machines. I don't believe xargs has an equivalent job management function.

Re: An Opinionated Guide to Xargs

#43

Post by bsmithers »

chubot wrote: post #16
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

I mention it here: https://www.oilshell.org/blog/2021/08/xargs.html#xargs-p-aut... What does it do that xargs and shell can't? (honest question)

Not to be pedantic, but that's a bit of a non-argument. _Of course_ you can do it with xargs and shell, but imho parallel is generally more convenient, especially for remote execution. It provides a higher level of abstraction for such tasks.

Re: An Opinionated Guide to Xargs

#44

Post by ViViDboarder »

lisper wrote: post #32

Note that the suggested: rm $(ls | grep foo) will not work if you have file names that contain spaces. Shell programming is planted thick with landmines like this.

The linked article doesn’t suggest this. They explicitly suggest against it.

> Besides the extra ls, the suggestion is bad because it relies on shell's word splitting. This is due to the unquoted $(). It's better to rely on the splitting algorithms in xargs, because they're simpler and more powerful.

Re: An Opinionated Guide to Xargs

#45

Post by chubot »

pwg wrote: post #37

Since the blog author is commenting here, you have this statement part way down your blog: > That is, grep doesn't support an analogous -0 flag. However, the GNU grep variant does have an analogous flag: -z, --null-data Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline. Like the -Z or --null option, this option can be used with commands like sort -z to pr…

Ah cool, I didn't know that! I'll update the blog post. (What a cacophony of flags)

Edit: It seems that grep -0 isn't taken for something else and they should have used it for consistency? The man page says it's meant to be used with find -print0, xargs -0, perl -0, and sort -z (another inconsistency)

Re: An Opinionated Guide to Xargs

#46

Post by kazinator »

In 2002, I implemented xargs in Lisp, in the Meta-CVS project.

It is quite necessary, because you cannot pass an arbitrarily large command line or environment in exec system calls.

Of course, this doesn't have the problem requiring -0 because we're not reading textual lines from standard input, but working with lists of strings.

  ;;; This source file is part of the Meta-CVS program,
  ;;; which is distributed under the GNU license.
  ;;; Copyright 2002 Kaz Kylheku

  (in-package :meta-cvs)

  (defconstant *argument-limit* (* 64 1024))

  (defun execute-program-xargs (fixed-args &optional extra-args fixed-trail-args)
    (let* ((fixed-size (reduce #'(lambda (x y)
                                   (+ x (length y) 1))
                               (append fixed-args fixed-trail-args)
                               :initial-value 0))
           (size fixed-size))
      (if extra-args
        (let ((chopped-arg ())
              (combined-status t))
          (dolist (arg extra-args)
            (push arg chopped-arg)
            (when (> (incf size (1+ (length arg))) *argument-limit*)
              (setf combined-status
                    (and combined-status
                         (execute-program (append fixed-args
                                                  (nreverse chopped-arg)
                                                  fixed-trail-args))))
              (setf chopped-arg nil)
              (setf size fixed-size)))
          (when chopped-arg
            (execute-program (append fixed-args (nreverse chopped-arg)
                                     fixed-trail-args)))
          combined-status)
        (execute-program (append fixed-args fixed-trail-args)))))

Re: An Opinionated Guide to Xargs

#47

Post by aaaaaaaaaaab »

fiddlerwoaroof wrote: post #22

I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

Some additional tips:

1. You don't need the parentheses.

2. If you use process substitution [1] instead of a pipe, you will stay in the same process and can modify variables of the enclosing scope:

    i=0
    while read -r v; do
        ...
        i=$(( i + 1))
    done
The drawback is that this way `do_something` has to come after `done`, but that's bash for you ¯\_(ツ)_/¯

[1] https://www.gnu.org/software/bash/manual/html_node/Process-S...

Re: An Opinionated Guide to Xargs

#49

Post by Zababa »

ineedasername wrote: post #17
Zababa wrote: post #5

Earlier quoted context omitted.

I've been thinking about titles, and it's hard to make a good one that doesn't look like a total cliché. "X considered harmful", "an opinionated guide to X", some kind of joke or reference, what could be a collection of tags (X, Y and Z), "things I have learned doing X", etc.

In this case a less cliche/click-baity title could simply be: "A Response to Xargs Criticism"

I think this title is fine, it's mostly that after spending some time on Hacker News all the titles start to look the same.

Re: An Opinionated Guide to Xargs

#50

Post by bialpio »

jmholla wrote: post #25
andy81 wrote: post #12

Today I appreciated Powershell

Can you expand on that? I've never had trouble leveraging xargs and find it aligns well with shell piping.

Not OP but to me the best thing about PowerShell is that it recognizes that text is not always the best way to output results from commands if you care about creating pipelines. In short, it passes objects around so there's no need for parsing text.
Post reply on HN