Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Design of GNU Parallel (2015)

gnu.org

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Design of GNU Parallel (2015)

#41

Post by [deleted] »

justeleblanc wrote: post #33
hexane360 wrote: post #28

Earlier quoted context omitted.

It's nonsense because the standard in academic settings is to cite works which contribute scientifically to the current work, not merely utilities. If I publish a paper on a command line tool for parallel processing, inspired by features from GNU parallel, I would cite GNU parallel. But if I'm doing (for instance) computational biology work, I'm not going to cite: - the Linux kernel - Python - Matlab - GNU parallel -…

Also, in what world is OSS financed by citations (which is stated as fact in the manpage)? The whole thing is just bizarre. Do I have to cite the manufacturer of my desk because I wrote my paper there?

[deleted]

Re: Design of GNU Parallel (2015)

#42

Post by fbdab103 »

imglorp wrote: post #26

Don't forget "make -j" is another option.

I was just attempting to parallelize a makefile (~500 files, ~20 minutes per file), and I was not happy with the experience. Make syntax for globbing is not ideal. Doubly so as my files had spaces inside of them. All solvable of course, but I feel more comfortable leaning on a parallel/xargs/find workflow than esoteric make syntax to handle the realities of filenames in the wild.

Which is a shame - 95% of my make usage is PHONY targets where I have a task and not a generated artifact. My current use case would have greatly benefited from the native parallelism and the ability to restart only failed files.

Re: Design of GNU Parallel (2015)

#43

Post by chungy »

RhysU wrote: post #39
hexane360 wrote: post #28

Earlier quoted context omitted.

It's nonsense because the standard in academic settings is to cite works which contribute scientifically to the current work, not merely utilities. If I publish a paper on a command line tool for parallel processing, inspired by features from GNU parallel, I would cite GNU parallel. But if I'm doing (for instance) computational biology work, I'm not going to cite: - the Linux kernel - Python - Matlab - GNU parallel -…

> It's nonsense because the standard in academic settings is to cite works which contribute scientifically to the current work, not merely utilities. Whether or not it's standard is irrelevant. Ole asked you to cite him if you use it. So, if you publish academically, either don't use it or cite him. If not using GNU Parallel hinders your science then the tool must be material to your work flows. For comparison, how m…

At least in the real world, free software doesn't demand that you agree with authors or do anything really. For as long as Ole keeps Parallel as free software, we can use it regardless of complying with requests.

Quite honestly, I think the behavior is on the highest order of jerkishness. A nice request could be done in the documentation, instead the path chosen is to bully users of the software.

Once more, because it is free software, we are free to use it despite what Ole thinks. We are free to patch it out too.

Re: Design of GNU Parallel (2015)

#44

Post by fbdab103 »

KronisLV wrote: post #11
michalc wrote: post #9

I've never been sure if it's too much of a hack, but I've used GNU parallel in Docker containers as a quick and easy way of getting multiple processes running for web applications. And with the `--halt now,done=1` option (that I think is relatively recent?) it means that if any of the parallel processes exit, parallel would exit itself, the whole container will shut down, and external orchestration would start anothe…

I've used Supervisor pretty successfully for this as well: http://supervisord.org/ Example of installing it in a Debian/Ubuntu container during container build, here's an example Dockerfile: RUN apt-get update \ && apt-get -yq --no-upgrade install \ supervisor \ && apt-get clean \ && rm -rf /var/lib/apt/lists /var/cache/apt/* Then it's possible to create a configuration file, for example /etc/supervisord.conf, to spe…

This is pretty crafty. I do not know supervisor well enough - if one of the services fail, can you engineer supervisor to also crash so that it would bubble up to the container infrastructure? My understanding is that standard supervisor would let the process die and/or restart the service.

Re: Design of GNU Parallel (2015)

#45

Post by attractivechaos »

fmajid wrote: post #31
ajsnigrutin wrote: post #24

Earlier quoted context omitted.

Yep, that's the great thing about perl... take a 20 year old script and it still works today. In comparison, if they used python, they'd be using python 2.2.

That's basically a side-effect of Perl being a dead language, frozen because Perl 6 will never happen. It's surprisingly hard to eradicate, however.

> That's basically a side-effect of Perl being a dead language

Keeping long-term backward compatibility does not necessarily mean dying. C is 50 years old and still alive. I have written a lot more Perl than Python. IMHO, Perl is dying because its syntax is arcane and confusing. We can't solve this problem unless we design a brand new language.

Re: Design of GNU Parallel (2015)

#46

Post by Alifatisk »

BooneJS wrote: post #30

Before GNU Parallel I used to use Ruby's workers and job queue to keep ${N} cores busy with work. It sorta worked like GNU parallel but was quite basic. I've since switched to using GNU Parallel. Stable code I don't have to write doesn't have to be maintained... not to mention it has more features than I normally supported.

What did you use exactly? I am curious, Resque? Sidekick?

Re: Design of GNU Parallel (2015)

#48

Post by uhtred »

fmajid wrote: post #31
ajsnigrutin wrote: post #24

Earlier quoted context omitted.

Yep, that's the great thing about perl... take a 20 year old script and it still works today. In comparison, if they used python, they'd be using python 2.2.

That's basically a side-effect of Perl being a dead language, frozen because Perl 6 will never happen. It's surprisingly hard to eradicate, however.

Perl 5 is actively developed still though, and presumable will become Perl 7 at some point.

Why does a language being stable mean it's dead? Is Awk dead?

Re: Design of GNU Parallel (2015)

#49

Post by hexane360 »

RhysU wrote: post #39
hexane360 wrote: post #28

Earlier quoted context omitted.

It's nonsense because the standard in academic settings is to cite works which contribute scientifically to the current work, not merely utilities. If I publish a paper on a command line tool for parallel processing, inspired by features from GNU parallel, I would cite GNU parallel. But if I'm doing (for instance) computational biology work, I'm not going to cite: - the Linux kernel - Python - Matlab - GNU parallel -…

> It's nonsense because the standard in academic settings is to cite works which contribute scientifically to the current work, not merely utilities. Whether or not it's standard is irrelevant. Ole asked you to cite him if you use it. So, if you publish academically, either don't use it or cite him. If not using GNU Parallel hinders your science then the tool must be material to your work flows. For comparison, how m…

"Ole asked you to cite him if you use it. So, if you publish academically, either don't use it or cite him."

Why? Whether something has contributed meaningfully to my research is my decision, not Ole's. Not having light "hinders my science", so I'll be sure to cite Edison on all my papers.

I agree with the sibling commentator that Ole's behavior is jerkish. Not because he asked for citations, but that he misleads users by claiming his request is standard, when it is decidedly not. He also obfuscates the voluntary nature of his request as much as possible, to make it seem like citing is a legal requirement. And he is inflammatory in responding to people who make the perfectly valid decision to not cite him, or to patch the notice out.

Re: Design of GNU Parallel (2015)

#50

Post by KronisLV »

fbdab103 wrote: post #44
KronisLV wrote: post #11

Earlier quoted context omitted.

I've used Supervisor pretty successfully for this as well: http://supervisord.org/ Example of installing it in a Debian/Ubuntu container during container build, here's an example Dockerfile: RUN apt-get update \ && apt-get -yq --no-upgrade install \ supervisor \ && apt-get clean \ && rm -rf /var/lib/apt/lists /var/cache/apt/* Then it's possible to create a configuration file, for example /etc/supervisord.conf, to spe…

This is pretty crafty. I do not know supervisor well enough - if one of the services fail, can you engineer supervisor to also crash so that it would bubble up to the container infrastructure? My understanding is that standard supervisor would let the process die and/or restart the service.

Supervisor allows you to have event listeners (e.g. for processes quitting/crashing), so you can use those to achieve that and kill supervisor itself. Here's an example of people doing just that: https://gist.github.com/tomazzaman/63265dfab3a9a61781993212f...
Post reply on HN