An Opinionated Guide to Xargs • Hacker News Board

Re: An Opinionated Guide to Xargs

#131

Post by michaelcampbell » Sun, Aug 22, 2021, 4:59 PM UTC

> I've used -P 32 to make day-long jobs take an hour! You can't do that with a for loop.

    for file in *; do
      command_using_file &
    done
    wait

?

I use variations on this all the time; pause while load is high, pause while 'x' or more things are running, sleep between invocations, etc.

It may not be as convenient for some cases, but "can't do that..." is not quite correct either.

The post is starting to feel like a hammer/nail argument, IMO.

Re: An Opinionated Guide to Xargs

#132

Post by michaelcampbell » Sun, Aug 22, 2021, 5:02 PM UTC

MainJane wrote: post #116

chubot wrote: post #59

Earlier quoted context omitted.

OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. Y…

Your `do_one`: * does not buffer stderr * does not check if the disk is full for a period of time during a task (thus risking incomplete output) * does not clean up, if killed * does not work correctly if task_with_stdout is a composed command Given that GNU Parallel is a drop-in replacement for xargs, I am curious why you find it a 'weirdo interface'.

Any sufficiently NIH'd can be considered weird. -- Not Isaac Asimov

Re: An Opinionated Guide to Xargs

#133

Post by yudlejoza » Mon, Aug 23, 2021, 12:29 AM UTC

patrickdavey wrote: post #98

yudlejoza wrote: post #83

Of xargs, for, and while, I have limited myself to while. It's more typing everytime but saves me from having to remember so many quirks of each command. cat input.file | ... | while read -r unit; do ${unit}; done | ... between 'while read -r unit' and 'while IFS= read -r unit' I can probably handle 90% of the cases. (maybe I should always use IFS since I tend to forget the proper way to use it).

Would you mind expanding with a couple of examples? (E.g. using "foo bar" as a single line or split by whitespace). I suspect I'll really like your way of doing things, but an example would be very handy.

The example of "foo bar" didn't work with while but inserting tr fixes it:

    echo "foo bar" | tr ' ' '\n' | while read -r var; do echo ${var}; done

For examples in general, I guess something like "cat file.csv" could work. (the difference between using IFS= and not using it is essentially whether we want to preserve leading and trailing whitespaces or not. If we want to preserve, then we should use IFS=).

Re: An Opinionated Guide to Xargs

#134

Post by cben » Mon, Aug 30, 2021, 6:38 AM UTC

Godel_unicode wrote: post #67

chubot wrote: post #59

Earlier quoted context omitted.

OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. Y…

Thank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python. For what it's worth, I consider oil to be closer…

The point of oil that there are really basic things like safe quoting that shells should do well, yet none of the posix shell do!

Functional: there are interesting shells like Elvish. But it really goes PowerShell by adding internal rich data pipelines that dont have a unixy stream-of-bytes representation. Oil does NOT go that way; it works on stuff like QSN to make pure unix interconnects more robust.

Re: An Opinionated Guide to Xargs

#135

Post by cben » Mon, Aug 30, 2021, 9:13 AM UTC

chubot wrote: post #45

pwg wrote: post #37

Since the blog author is commenting here, you have this statement part way down your blog: > That is, grep doesn't support an analogous -0 flag. However, the GNU grep variant does have an analogous flag: -z, --null-data Treat the input as a set of lines, each terminated by a zero byte (the ASCII NUL character) instead of a newline. Like the -Z or --null option, this option can be used with commands like sort -z to pr…

Ah cool, I didn't know that! I'll update the blog post. (What a cacophony of flags) Edit: It seems that grep -0 isn't taken for something else and they should have used it for consistency? The man page says it's meant to be used with find -print0, xargs -0, perl -0, and sort -z (another inconsistency)

See https://github.com/fish-shell/fish-shell/issues/3164#issueco... for incomplete but large survey of NUL flags.

Takeaways: (1) There is no consistency in flag names, even --long ones (2) impressively many tools do support it! Note that some affect only input or only output. (3) All do NUL-terminated, not NUL-separated. That's fortunate — matches \n usage, and gives distinct representations for [] vs [""].