Earlier quoted context omitted.
Oh man, parallel is awful. I have to rant about this because I literally tried it again today morning. Every single damn time I try parallel and decide to give it another chance, something ends up not working or causing a problem. I can never get it to just do what I want and get out of the way. Today I foolishly thought maybe I was the one who was holding it wrong every single time in the past, so I copy pasted anot…
> Today I foolishly thought maybe I was the one who was holding it wrong […] Apparently goes on to describe being confused about parallel reading from the standard input?
I wrote an bash enumerator because I was sick of xargs
81–90 of 202 posts
Re: I wrote an bash enumerator because I was sick of xargs
#82[dead]
Re: I wrote an bash enumerator because I was sick of xargs
#83Earlier quoted context omitted.
Depending on the actual command, this can be far slower and less efficient than xargs. You're creating a separate process for each invocation of bar when a lot of commands will take many targets for a single invocation. Try this with find and grep vs xargs. There's a big difference.
3 reasons against that: * in a lot of cases the performance just doesn't matter * xargs gets you the spaces in filenames landmine * some commands don't even support multiple target filename arguments For scripts in long term use, yeah, sure, figure out xargs maybe. Any other situation with a "| while read" solution, especially on an interactive session, is an oddball and simplicity wins.
xargs supports null termination.
While it's true not every command supports multiple targets, presumably you know if you're using one of those commands.
Re: I wrote an bash enumerator because I was sick of xargs
#84On the topic of xargs replacements, I love gnu parallel. The --dry-run flag of parallel made me confident to do more batch processing than I ever did with xargs. Parallel has an option for almost everything, it's almost too much. But I have shopped around for alternatives. The creator Ole Tange maintains a painstakingly long article of the alternatives and their differences. [0] The gnu parallel book and reading mate…
[deleted]
Re: I wrote an bash enumerator because I was sick of xargs
#85I use: find . -name '*.log' -print0 | xargs -0 rm For this simple example (derived from the article), find also has a delete operator. This null termination is now a POSIX standard.
[deleted]
Re: I wrote an bash enumerator because I was sick of xargs
#86Good idea but strange syntax. It would be more idiomatic if the command came first and the list of globs last. Additionally the use of "--" is not what everybody expects: here it is used to introduce one argument, the command, while it's usually meant to introduce multiple arguments without worrying if they have a leading "-". A possible revised syntax with command as the first argument followed by a list of globs op…
I thought -- was typically a separator for passing a list of arguments directly to some subcommand
The historical meaning is, "do not interpret command line arguments following --"; e.g. the classical form shown by `echo > -f; rm -- -f`. Git has a more complex interpretation of it and no doubt there's others, but this root interpretation remains generally sound: It acts as a boundary between 'complex and intelligent processing of @ARGV elements' and 'every remaining element of @ARGV after -- is treated a string literal without further processing'.
Re: I wrote an bash enumerator because I was sick of xargs
#87in zsh you can just write: for x (*.sh); echo "before $x after" or for x in *.sh; echo "before $x after" or for x (*.sh) { echo -n "before "; echo -n $x; echo " after" }
missed the point. reason to use xargs or parallel are generally two: list of arguments is too long, or list of arguments that is kept in memory would take too much for example for a in `find / ` ; do echo $a ; done will take A LOT of memory, while using find's -exec, xargs, or parallel will not
In my experience once you get to the point where you run out of space in the glob you’re often suffering with poor performance from spawning children as well and it’s time to move to a more formal program even if it’s a script, writing out work plans and completions to list files to avoid wasted time. It’s often a great guardrail to remind you to do this
Re: I wrote an bash enumerator because I was sick of xargs
#88Earlier quoted context omitted.
> the app just hung there trying to figure out how long its command line can be? That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. The citation thing is a little silly, I'll give you that one.
> That's common on linux. Many tools read from stdin if a file path isn't given: cat, xargs, base64, cksum, etc. No. I did pipe to stdin. It's not my first time using Linux... Here's a command line I ran right now , and the output I see: $ echo "http://www.example.com" | parallel -k -j 8 curl -s "{}" Academic tradition requires you to cite works you base your article on. [...more nonsense...] To silence this citation…
> parallel: Warning: Finding the maximal command line length. This may take up to 1 minute.
> So I wait a few seconds [snip]
This warning is only ever printed if running in Cygwin, not Linux or macOS or elsewhere. Cygwin is notoriously slow.
# This is slow on Cygwin, so give Cygwin users a warning
if($^O eq "cygwin" or $^O eq "msys") {
::warning("Finding the maximal command line length. ".
"This may take up to 1 minute.")
}
Re: I wrote an bash enumerator because I was sick of xargs
#89> Or maybe you pipe into xargs and pray your filenames don’t have spaces… Most of this, if not all, is fixable by adding a `export IFS=$'\n'` to your bashrc. I'm not trying to disregard your project, just point out something that took me years to learn and I currently use extensively to solve this very problem. Perhaps you didn't know about it until now... :)
Or, you use `xargs -0` for null termination instead of white space termination. `find` conveniently supports `-print0` that will use null character as separator.
Re: I wrote an bash enumerator because I was sick of xargs
#90find -print0...|xargs -0...
works for me, and i don't always want to execute something and xargs can run parallel processes. i feel like this guy never bothered to read the man page.