Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

211–220 of 277 posts

Re: GNU Parallel, where have you been all my life?

#211

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

1) Amdahl's law means it's not useful to have hundreds of cores for general purpose computing. There's not that much parallel work to do in typical applications. Increasing the proportion of work that's parallelizable for a given application pays dividends when you have more cores - that's why Servo is so exciting. In some cases, picking an O(n2) algorithm that's easy to parallelize will be faster than a less parallizable O(nlog(n)) algorithm - this is true for problems like Single-Source Shortest Paths (SSSP).

2) Shared resources (in-memory mutable data, hardware devices) mean the ratio of contention to CPU work goes up when you have more cores.

3) Cores on a single die need to share the same constraints - thermal limits and transistor count. So you're best off having enough powerful cores to get you to a sweet spot of single-core performance vs multi-core parallelism.

4) It's hard to provide a performant and useful many-core machine model. Cache coherence makes it easier to program a many-core machine, but limits performance. Without it, you're stuck with distributed systems-style problems.

Re: GNU Parallel, where have you been all my life?

#212
post #81

Earlier quoted context omitted.

> Is the author still adding the "cite me or pay 10000€" notice to the output? And calling that GPL? IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works. > These are just some of the reasons I’ll never use parallel. Hey I’ve actually ranted on HN be…

> it’s not tradition to write citations for tools used to conduct research. Thia is true, but it also makes it very hard for academics and PhD students who mainly write software over papers. They get no citations and eventually have to leave academia. If we had a better practice of citing central software we use - at least the academic software that wants to be cited - we could have a more flourishing ecosystem of su…

Like a colophon for research?

Re: GNU Parallel, where have you been all my life?

#213
post #198

Earlier quoted context omitted.

That is a weird request, a source doesn’t need to be published in a journal to be cite-able. On the other hand, if you put a bibtex snippet on your site that indicates how you’d like to be cited, that is super helpful.

I wasn't very clear in my comment. I think the idea was that if the thing they were citing was a journal article, then the citations would actually mean something, as widely-cited journal articles are one of the currencies of most academic fields. While one certainly can cite an unpublished document, the author doesn't necessarily gain much from it in terms of their academic CV. If someone writes a piece of software…

I agree on all points, just trying to look at it from the other party’s point of view. But yeah, the situation with this sort of stuff is a pain.

There should be a high-prestige “journal of READMEs and User Handbooks,” haha.

Re: GNU Parallel, where have you been all my life?

#214

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

Have you considered finding a Connection Machine?

Re: GNU Parallel, where have you been all my life?

#215

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

Read a lot of this kind of post. Years ago I recall someone bleating for 8 cores when 1 or 2 was the norm. Now you want 256. Next generation will ask for thousands. All for nothing because you have no idea what to do with it except give the handwaviest justifications. A computer's a tool to do an actual job. You can and probably do have more computing power on your desktop than all the world's supercomputers put together from the 1970's.

https://en.wikipedia.org/wiki/Cray_X-MP

   Price US$7.9 million in 1977 (equivalent to $38.2 million in 2022)
   Weight 5.5 tons (Cray-1A)
   Power 115 kW @ 208 V 400 Hz[1]
   CPU 64-bit processor @ 80 MHz[1]
   Memory 8.39 Megabytes (up to 1 048 576 words)[1]
   Storage 303 Megabytes (DD19 Unit)[1]
   FLOPS 160 MFLOPS
In 2070 it still won't be enough for you. It never will be enough.

Re: GNU Parallel, where have you been all my life?

#216
post #153

Earlier quoted context omitted.

Software cannot be distributed with a clickwrap agreement under the GPL. Requiring the user to affirmatively agree to a contract is a clickwrap agreement even if the terms are non-monetary. The old “you are making a second agreement, not the one the software is distributed under” approach. Notionally the GPL allows you to disregard this but it may or may not be binding depending on your jurisdiction, and it’s certain…

It's not a clickwrap agreement, since the user doesn't agree to anything: To silence this citation notice: run 'parallel --citation' once.

did you ignore all the stuff before that about the implication of using that option and what the user agrees to by using it?

this is like saying that a user doesn't actually agree to anything just because they clicked "accept" in a EULA. you're just clicking buttons in software, it doesn't obligate you to anything!!! but actually yes that is most likely fairly binding in a lot of jurisdictions.

that is, again, literally the definition of a clickwrap licensing agreement and you cannot do that in GPL software, even if it's non-monetary. Requiring the user to submit a selfie in a funny hat would not be permissible under the GPL either. You can't limit what the user does with the software and how, or else it's not GPL.

it's open and shut, clickwrap agreements completely subvert and nullifies the moral stand the FSF is trying to make. And it doesn't matter how innocuous it seems, it undermines the whole point of the exercise.

fortunately the GPL includes a "severability" clause that basically allows you to ignore this and grants you a license regardless. but it is not a good look, it is not good behavior, and if every GPL'd package started adding random clickwrap agreements with big "IM A DOODOO HEAD IF I IGNORE THIS" parameters the whole ecosystem would degrade.

Arch and others are not only allowed but actually morally and practically in the right for stripping these messages, and it doesn't reflect well on Ole at all that he then goes on and throws more tantrums because he doesn't like the consequence of the license he chose.

If he wants to go proprietary, or BSD (which requires acknowledgement!), that's fine, but he's being a child and the terms he are adding are utterly uncompliant with GPL, and it's unprofessional for FSF to even humor him on this. If there were a hundred Oles the FSF would have a real problem on its hands, it's only because he's N=1 jerk that this is remotely tolerable.

Re: GNU Parallel, where have you been all my life?

#217
post #147

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

Perl6/Raku is my personal choice (when I can).

Python to me, is too far away from shell/unix. It is a programming language for writing applications. For the use case of writing shell scripts but in a more powerful language, perl is still the king here (or it should be. Sadly it doesn't appear to be the case. No one is using it except for die hard gray beards.)

Raku is a modern (still a big) language with kitchen sink. Again doesn't appear to be much uptake.

Re: GNU Parallel, where have you been all my life?

#218
post #171

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

GNU Parallel is also based on perl, so the footprint is quite large. GNU xargs implements limited parallelization, and is compiled C. This functionality is present within busybox, including the Windows version. https://www.linuxjournal.com/content/parallel-shells-xargs-u... GNU Parallel will have much greater functionality, but it will not reach as far as xargs.

> GNU Parallel is also based on perl

Time to rewrite it in Rust /s

:p

Re: GNU Parallel, where have you been all my life?

#219
post #189
post #128

Earlier quoted context omitted.

> and the citation notice is a one-time thing you can silence permanently This doesn't scale. Imagine if all the software you used nagged you and had their own individual methods to silence them. I don't think this would be reasonable. What makes this particular software so special?

> This doesn't scale. --will-cite

sure, but the other guy wants me to submit a picture in a funny hat, not a citation. and the third guy wants me to add some additional legal provisions and disclaimers to the GPL license.

you need a more generalized --clickwrap-consent parameter really. One that just says "whatever it is, I accept and I'll do it".

And that's exactly the thing GPL was supposedly founded to get away from. Restrictions on user freedoms. Especially violations so routine and tedious that we open-palm-slam "accept" without reading them.

You could absolutely write this to not look like a clickwrap agreement and lean on users. "please cite me, I'm an academic and impact matters" in the manfile or --help is not something anyone would ever get upset about or probably even patch to remove.

The only reason it's OK is because basically everyone knows it's not enforceable because of the severability part of the GPL. But it's blatantly designed to look like a serious and enforceable notice to users who don't know that, and require affirmative action from the user to "consent" and bypass the screen. And clickwrap agreements of this type are generally enforceable if there is not something like the GPL that allows you to ignore it.

like I flatly do not get why this is even debatable or questionable, the dude is trying to pull a fast one on users with a scary-sounding legal notice that implies that you need to accept this clickwrap agreement. and it's not entirely clear that he cannot actually burden you with this in all jurisdictions, since it's an agreement between you and the author that exists outside the actual source code/distribution. You can end up paying for free stuff in lots of places in life, if you're not aware about what "should" be free, and those agreements stand and are enforceable even though the thing was supposed to be free. You agreed to it. You don't have to, the GPL says that, you can edit the software to remove it without consenting, but you did accept it.

Letting the camel's nose under the tent on clickwrap agreements on GPL'd software is such an incredibly bad idea legally and morally, and this dude has been an utter dick about anyone who questions that. Sure, "he's willing to do it and nobody else is stepping up" but on the other hand he's also going off and attacking other maintainers doing their jobs, too. But that's not Stallman's problem I guess. That's another problem that only works with N=1 jerk, if that was normalized we'd have a problem.

I do not get why this guy is getting this special blessing or dispensation from FSF. Like it's not just that he's a random weirdo releasing under GPL and then trying to add additional terms (lol get stuffed), this is all occurring with the FSF's blessing, Stallman's signoff, and in the GNU distribution. Official GNU clickwrap license I guess.

At the end of the day - if the guy can't be satisfied with a polite request in the manfile, wow that sucks. But the GPL isn't about you, it's about the end user. There are explicitly licenses like BSD that require acknowledgement if that's your thing!

Re: GNU Parallel, where have you been all my life?

#220
post #216

Earlier quoted context omitted.

It's not a clickwrap agreement, since the user doesn't agree to anything: To silence this citation notice: run 'parallel --citation' once.

did you ignore all the stuff before that about the implication of using that option and what the user agrees to by using it? this is like saying that a user doesn't actually agree to anything just because they clicked "accept" in a EULA. you're just clicking buttons in software, it doesn't obligate you to anything!!! but actually yes that is most likely fairly binding in a lot of jurisdictions. that is, again, litera…

It doesn't look like it obligates me to do anything. It contains a request to cite, not a requirement:

  If you use programs that use GNU Parallel to process data for an article in a
  scientific publication, please cite:
  
    Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин').
    Zenodo. https://doi.org/10.5281/zenodo.8175685
  
  This helps funding further development; AND IT WON'T COST YOU A CENT.
  If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
  
  More about funding GNU Parallel and the citation notice:
  https://www.gnu.org/software/parallel/parallel_design.html#citation-notice
  
  To silence this citation notice: run 'parallel --citation' once.
[edit]

This is really similar to the kerfuffle where the maintainer of Home Assistant asked distros to not repackage HA; it's a request, a bit at odds with community norms for libre software, but one that people are legally free to ignore.

Post reply on HN