Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

191–200 of 277 posts

Re: GNU Parallel, where have you been all my life?

#191

Post by JonChesterfield »

zackmorris wrote: post #178

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

This exists now. Some AI accelerators are a grid of independent compute units with their own memory, message passing between them. Graphcore's IPU is an instance.

An AMD GPU is a grid of independent compute units on a memory hierarchy. At the fine grain, it's a scalar integer unit (branches, arithmetic) and a predicated vector unit, with an instruction pointer. Ballpark of 80 of those can be on a given compute unit at the same time, executed in some order and partially simultaneously by the scheduler. GPU has order of 100 compute units, so that's ~8k completely independent programs running at the same time.

You've got a variety of programming languages available to work with that. There's a shared address space with other GPUs and the system processors, direct access to system and GPU local memory. Also some other memory you can use for fast coordination between small numbers of programs.

There's a bit of a disconnect between graphics shaders, the ROCm compute stack and what you can build on the hardware if so inclined. The future you want is here today, it just has a different name to what you expected.

Re: GNU Parallel, where have you been all my life?

#192

Post by 542458 »

dahart wrote: post #81
throwaway277432 wrote: post #33

Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers. These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need…

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> IIRC the citation notice was cleared by Stallman as GPL compatible

Do you have a source for this? Im confused by this, as the GPL section 7 is pretty clear that additional restrictions are effectively void. I suppose it’s technically not contrary to the GPL to idly state those restrictions, but it is contrary to the GPL to expect them to do anything. If the author is deliberately including an impotent clause in the hope that people will follow it anyways, I feel that trying to confuse or scare people into doing something the GPL gives them explicit permission to do is contrary to the spirit of the GPL.

Furthermore, trying to retaliate against people who (as permitted by the GPL) remove the citation notice, as the author here has done, seems very contrary to the spirit of the GPL.

Re: GNU Parallel, where have you been all my life?

#193

Post by dahart »

bee_rider wrote: post #185
foldr wrote: post #101

Earlier quoted context omitted.

> it’s not tradition to write citations for tools used to conduct research Academics seem to have a very blinkered attitude to this. I wrote some software that was popular for a while in a niche field, and people were forever asking me to waste my time by 'publishing' the manual in some pointless journal so that they could cite something and give me credit. Writing useful software counts for less in that world than p…

That is a weird request, a source doesn’t need to be published in a journal to be cite-able. On the other hand, if you put a bibtex snippet on your site that indicates how you’d like to be cited, that is super helpful.

Ole did provide a BibTeX entry to a USENIX magazine article about Parallel, which is fine, though I was always taught that non-journal references generally belong in footnotes or an appendix and not the bibliography, especially for something you’re not referencing for research purposes. Not sure if footnote or appendix or open-source usage citations count for what Ole needs; I’d guess he wants citations you can easily index using Google Scholar or other citation indexes, i.e. it should count toward Parallel’s H-index (https://en.wikipedia.org/wiki/H-index)

Re: GNU Parallel, where have you been all my life?

#194

Post by cb321 »

astrodust wrote: post #117
Aissen wrote: post #26

GNU parallel is great for the kind of tasks highlighted in the post. Note that being written in Perl, it's slower than its simpler C counterpart moreutils parallel. And that in many uses cases xargs --max-procs=$(nproc) can replace it.

`xargs` has you covered in more cases than most realize.

This really is true and you may be understating with "most". Here are a couple:

    mkdir /tmp/g
    seq 1 10 | tr \\n \\0 |
      xargs -0n2 -P4 bash -c 't=$EPOCHREALTIME; sleep $((RANDOM%5)); echo "$@" >/tmp/g/$t' d0
    cat /tmp/g/*
Another one is

    xargs -P "$(nproc)" --process-slot-var=s sh -c 'grep X "$@" >>/tmp/g.$s' d0
    cat /tmp/g.*
You can also cobble together that second style with a custom config setup wherein a command is given $s and responds with some host names and there might be an `ssh` in front of the `grep`, for example. That `d0` argument (for $0) is a bit janky and there can be shell quoting issues, of course. But then again, you may not have hostile filenames/whatever. Remote loadavg adaptation might be nice, but then again, maybe you control all the remotes. Similarly, I could not get back-to-back executions of the EPOCHREALTIME thing closer than 250 microseconds. So, collision basically will not happen even though it probably could in theory.

Re: GNU Parallel, where have you been all my life?

#195

Post by MawKKe »

zackmorris wrote: post #178

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

1000 cores?? I don't have 100 cores! What do you even need 10 cores for? Well, here's 4 cores. Give 2 to your brother. Don't go wasting all those hyper threads all at once!

Intel ca. 2010, probably

Re: GNU Parallel, where have you been all my life?

#196

Post by dahart »

542458 wrote: post #192
dahart wrote: post #81

Earlier quoted context omitted.

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> IIRC the citation notice was cleared by Stallman as GPL compatible Do you have a source for this? Im confused by this, as the GPL section 7 is pretty clear that additional restrictions are effectively void. I suppose it’s technically not contrary to the GPL to idly state those restrictions, but it is contrary to the GPL to expect them to do anything. If the author is deliberately including an impotent clause in the…

Here’s one source, there are several other places Ole has talked about it: https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...

I think the confusing issue here is that the notice is not a license requirement, it does not add additional licensing restrictions. It’s an honor-system agreement between the user and Ole, and does not involve the GPL. It does seem to be walking a very fine line, and it’s easy for users to not understand the distinction, but I believe the notice does adhere to the GPL’s rules, even if it doesn’t initially appear to for us non-lawyers.

Re: GNU Parallel, where have you been all my life?

#197

Post by dehrmann »

electroly wrote: post #116
Matl wrote: post #60

Earlier quoted context omitted.

If you're a startup and relicensing previously open source code under a restrictive license or doing other shady things you'll get plenty of defenders to line up to say 'hey, they have to make a living somehow', but if a single guy tries to make a living via a simple message in a widely used program all hell breaks loose. What gives?

Notably, GNU Parallel did not relicense; it's still GPL. The author wants to have his cake (gain the popularity benefits of being a GPL-licensed GNU tool, be able to carpetbomb Stack Overflow with "use GNU Parallel" answers, etc.) and eat it too (get people to cite or pay him as a condition of using the product). Since this isn't possible (GPL doesn't allow additional restrictions), but the author still really wants…

So if it's GPL, I can remove the citation bits, release it under the GPL as free-parallel, and everything's OK?

Re: GNU Parallel, where have you been all my life?

#198

Post by foldr »

bee_rider wrote: post #185
foldr wrote: post #101

Earlier quoted context omitted.

> it’s not tradition to write citations for tools used to conduct research Academics seem to have a very blinkered attitude to this. I wrote some software that was popular for a while in a niche field, and people were forever asking me to waste my time by 'publishing' the manual in some pointless journal so that they could cite something and give me credit. Writing useful software counts for less in that world than p…

That is a weird request, a source doesn’t need to be published in a journal to be cite-able. On the other hand, if you put a bibtex snippet on your site that indicates how you’d like to be cited, that is super helpful.

I wasn't very clear in my comment. I think the idea was that if the thing they were citing was a journal article, then the citations would actually mean something, as widely-cited journal articles are one of the currencies of most academic fields. While one certainly can cite an unpublished document, the author doesn't necessarily gain much from it in terms of their academic CV.

If someone writes a piece of software specifically for the purposes of doing certain types of scientific research, and then other scientists use this software to help conduct published experiments, then IMO it really ought to be possible to give that person meaningful credit for their work. It's a perfectly legitimate way to contribute to a field, even if it does not take the form of a paper. But with the system as it stands, the only way to get meaningful credit is to publish a pointless paper saying, in effect, "Hey! I wrote some software!"

>if you put a bibtex snippet on your site that indicates how you’d like to be cited, that is super helpful.

I should probably have done that, but from my point of view it didn't really matter. I have a name, and the software had a website. I didn't really mind exactly how individual people chose to cite it. The absence of a ready-baked bibtex snippet would never be accepted as an excuse for failing to cite any other kind of source.

Re: GNU Parallel, where have you been all my life?

#199

Post by cstrahan »

Aissen wrote: post #26

GNU parallel is great for the kind of tasks highlighted in the post. Note that being written in Perl, it's slower than its simpler C counterpart moreutils parallel. And that in many uses cases xargs --max-procs=$(nproc) can replace it.

I also recommend checking out `xe`: https://github.com/leahneukirchen/xe

It’s like xargs with sane defaults and a couple tricks of its own.

Re: GNU Parallel, where have you been all my life?

#200

Post by mongol »

lysium wrote: post #147
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

I once read a HN thread that recommended Go for this, and it made me interested. I think it was a useful suggestion, it made me learn Go, and I kind of agree with it, 5+ years after. It is not a shell, but it is simple and fast and useful.
Post reply on HN