Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Opinionated Guide to Xargs

oilshell.org

Post reply on HN
Print view View original

101–110 of 135 posts

Re: An Opinionated Guide to Xargs

#101

Post by andy81 »

bialpio wrote: post #56
bialpio wrote: post #50

Earlier quoted context omitted.

Not OP but to me the best thing about PowerShell is that it recognizes that text is not always the best way to output results from commands if you care about creating pipelines. In short, it passes objects around so there's no need for parsing text.

Two examples from the article translated into PS (sorry, I'm a bit rusty so the second one may not be the shortest possible): PS> "alice", "bob" | echo PS> Get-ChildItem . -Include "*test.cpp","*test.py" -Recurse | foreach { Remove-Item $_.Name } No text parsing in sight, and the object attributes can be tab-completed from the shell (e.g. I tab-completed the `$_.Name`).

Thanks, we were thinking of the same thing.

Re: An Opinionated Guide to Xargs

#102

Post by MonkeyClub »

phone8675309 wrote: post #21
legobmw99 wrote: post #2

This is only tangentially related, but after all the posts here the last few days about thought terminating cliches, I can’t help but reflect on the “X considered harmful” title cliche

What every X should know about Y, an opinionated take on Z considered harmful

...with an example Lisp implementation written in APL translating into 6502 assembly :)

Re: An Opinionated Guide to Xargs

#103

Post by scottlamb »

chubot wrote: post #59
comex wrote: post #27

Earlier quoted context omitted.

One thing parallel can do better than xargs is collect output. If you use `xargs -P`, all processes share the same stdout and output may be mixed arbitrarily between them. (If the program being executed uses line buffering, lines usually won't be mixed together from multiple invocations, but they can be if they're long enough). In contrast, `parallel` by default doesn't mix together output from different commands at…

OK thanks, looks like there are several features of GNU parallel that users like. For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file: do_one() { task_with_stdout > $dir/$task_id.txt } So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them. Y…

A lot of this comes down to familiarity. I tend to use "make -j 100" for what you're describing. If I write the Makefile carefully [1], it will handle resuming a half-finished job. I just looked and GNU parallel has a --resume argument which probably does something similar, and maybe with less hassle. But I don't do this often enough—and/or GNU parallel isn't "better enough"—that I'm likely to ever invest the time to learn GNU parallel.

btw, oil looks very cool. I hate how many footguns are in common shells.

[1] eg writing to a tempfile and atomically renaming into place: "task_with_stdout > $dir/task_id.txt.tmp && mv $dir/task_id.txt{.tmp,}"

Re: An Opinionated Guide to Xargs

#104

Post by scottlamb »

yudlejoza wrote: post #83

Of xargs, for, and while, I have limited myself to while. It's more typing everytime but saves me from having to remember so many quirks of each command. cat input.file | ... | while read -r unit; do ${unit}; done | ... between 'while read -r unit' and 'while IFS= read -r unit' I can probably handle 90% of the cases. (maybe I should always use IFS since I tend to forget the proper way to use it).

That way will bite you when the tasks in question are cheaper than fork+exec. There was a thread just the other day in which folks were creating 8 million empty files with a bash loop over touch. But it's 60X faster (really, I measured) to use xargs, which will do batches (and parallelism if you tell it to).

https://news.ycombinator.com/item?id=28192946

Re: An Opinionated Guide to Xargs

#105

Post by Arnavion »

bialpio wrote: post #56
bialpio wrote: post #50

Earlier quoted context omitted.

Not OP but to me the best thing about PowerShell is that it recognizes that text is not always the best way to output results from commands if you care about creating pipelines. In short, it passes objects around so there's no need for parsing text.

Two examples from the article translated into PS (sorry, I'm a bit rusty so the second one may not be the shortest possible): PS> "alice", "bob" | echo PS> Get-ChildItem . -Include "*test.cpp","*test.py" -Recurse | foreach { Remove-Item $_.Name } No text parsing in sight, and the object attributes can be tab-completed from the shell (e.g. I tab-completed the `$_.Name`).

You don't need to `foreach { Remove-Item $_.Name }` because Remove-Item can take the objects returned by Get-ChildItem directly.

Also, expanding the regex into `-Include` parameters is somewhat cheating since `-Include` only takes globs, and it just so happens that that particular regex can be converted into globs.

The general equivalent is:

    gci -re | ?{ $_.Name -match '.*_test\.(py|cc)' } | ri
(I used the shorter aliases because someone will probably read yours and reinforce the stereotype that PS is overly verbose.)

Re: An Opinionated Guide to Xargs

#106

Post by nerdponx »

orhmeh09 wrote: post #68
WhatIsDukkha wrote: post #15

I tend to reach for gnu parallel instead of xargs - https://www.gnu.org/software/parallel/parallel_alternatives.... parallel is probably on the complex side but its also been actively developed, bugfixed and had a lot of road miles from large computing users.

The nagware prompts of parallel are so objectionable that I will do a lot of things to avoid using it at all. So pretentious!

On the contrary, I think more FOSS authors should do things like this. Freedom doesn't mean you don't get to take credit for your work.

Re: An Opinionated Guide to Xargs

#107

Post by rcpt »

chubot wrote: post #31
rcpt wrote: post #23

awk '{ print your_command }' | bash Never can remember all the -I stuff around xargs

This is like the sed|bash anti-pattern mentioned in the original post, and quoted in the appendix on shell injection. I wouldn't say "never use it", but I would hesitate to ever put it in a script, vs. doing a one-off at the command line.

You don't pipe to bash on the first run. Use awk/sed without piping to workshop your commands. Once you've got them right send over to bash.

This is far superior to futzing with xargs interactive or whatever dry run feature they have.

Re: An Opinionated Guide to Xargs

#108

Post by bialpio »

Arnavion wrote: post #105
bialpio wrote: post #56

Earlier quoted context omitted.

Two examples from the article translated into PS (sorry, I'm a bit rusty so the second one may not be the shortest possible): PS> "alice", "bob" | echo PS> Get-ChildItem . -Include "*test.cpp","*test.py" -Recurse | foreach { Remove-Item $_.Name } No text parsing in sight, and the object attributes can be tab-completed from the shell (e.g. I tab-completed the `$_.Name`).

You don't need to `foreach { Remove-Item $_.Name }` because Remove-Item can take the objects returned by Get-ChildItem directly. Also, expanding the regex into `-Include` parameters is somewhat cheating since `-Include` only takes globs, and it just so happens that that particular regex can be converted into globs. The general equivalent is: gci -re | ?{ $_.Name -match '.*_test\.(py|cc)' } | ri (I used the shorter al…

Thanks, this is definitely closer to the original use of `egrep`! As for aliases, I prefer long forms because I don't need to think what the seemingly random collections of letters mean, and tab-completion / PowerShell ISE makes it mostly a non-issue when writing.

Re: An Opinionated Guide to Xargs

#109

Post by thayne »

fiddlerwoaroof wrote: post #22

I frequently find myself reaching for this pattern instead of xargs: do_something | ( while read -r v; do . . . done ) I’ve found that it has fewer edge cases (except it creates a subshell, which can be avoided in some shells by using braces instead of parens)

creating a subshell can lead to some surprising behavior if you aren't careful though.

Re: An Opinionated Guide to Xargs

#110

Post by michaelsbradley »

blibble wrote: post #63
jeffbee wrote: post #40

Earlier quoted context omitted.

parallel doesn't either, it just nags. I agree about how silly and annoying it is. Imagine if every time the parallel author opened Firefox he got a message reminding him to personally thank me if he uses his web browser for research, or if every time his research program calls malloc he has to acknowledge and cite Ulrich Drepper. Very very silly. Parallel is the better tool but the nagware impairs its reputation.

or every time a process called fork() you had to read some stupid message

echo will cite | parallel --bibtex
Post reply on HN