Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

171–180 of 277 posts

Re: GNU Parallel, where have you been all my life?

#171

Post by chasil »

BoppreH wrote: post #107

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

GNU Parallel is also based on perl, so the footprint is quite large.

GNU xargs implements limited parallelization, and is compiled C. This functionality is present within busybox, including the Windows version.

https://www.linuxjournal.com/content/parallel-shells-xargs-u...

GNU Parallel will have much greater functionality, but it will not reach as far as xargs.

Re: GNU Parallel, where have you been all my life?

#172

Post by mplewis »

lysium wrote: post #147
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

I use Go. You can run scripts with go run directly, and this package makes shell tasks easy: https://github.com/bitfield/script

Re: GNU Parallel, where have you been all my life?

#173

Post by memefrog »

dahart wrote: post #165
grotorea wrote: post #134

Earlier quoted context omitted.

But legally, does intention matter in either copyright or contract law? And specially someone who is neither licensee or licensor?

I’m not sure I understand what you’re asking. Why is the intention of the author of the license in question? In this case, Stallman simply clarified that Parallel’s notice did not count as a legal requirement and does not conflict with the GPL. His opinion wasn’t necessary, but since he wrote the license, it is authoritative. In this case, the question wasn’t brought to court, it was simply a clarifying discussion, a…

>His opinion wasn’t necessary, but since he wrote the license, it is authoritative.

No it isn't. Licences, like most legal documents, are construed objectively. The subjective intention of the author is totally irrelevant to the meaning.

Re: GNU Parallel, where have you been all my life?

#174

Post by memefrog »

dahart wrote: post #150
dec0dedab0de wrote: post #145

Earlier quoted context omitted.

Okay that makes sense. You're saying that since the GPL itself is not open, that it needs Stallman's approval for modifications that are not explicitly allowed. And I was saying that it does not necessarily mean those modifications are enforceable between two parties in a random jurisdiction, which comes down to courts and whatnot.

Yes kind-of… in this case Parallel’s notice is not a modification of the license at all, and Stallman is the person who ruled on this question and confirmed this to be true. The GPL doesn’t prevent authors from including a notice, and having a notice doesn’t conflict with the terms of the GPL. I feel like the whole problem here is that the legality of Parallel’s notice, and the separation of the notice from the GPL,…

>in this case Parallel’s notice is not a modification of the license at all

This is a question of law that only a court can answer.

Re: GNU Parallel, where have you been all my life?

#175

Post by [deleted] »

gorjusborg wrote: post #132
thomasahle wrote: post #97

Earlier quoted context omitted.

> it’s not tradition to write citations for tools used to conduct research. Thia is true, but it also makes it very hard for academics and PhD students who mainly write software over papers. They get no citations and eventually have to leave academia. If we had a better practice of citing central software we use - at least the academic software that wants to be cited - we could have a more flourishing ecosystem of su…

> They get no citations and eventually have to leave academia. You're welcome? Seriously though, adding the citation nag to software is two wrongs not making a right. As a software user, it isn't my fault academia hasn't figured out how to reward software contribution. If they can't figure it out, finding a greener pasture makes a lot of sense.

[deleted]

Re: GNU Parallel, where have you been all my life?

#176

Post by runeks »

coliveira wrote: post #127
BoppreH wrote: post #107

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

If doing even simple things requires looking up documentation, why does it matter whether the shell script is long or short?

Spending extra time doing simple things — because you need to Google e.g. "how to pass multiple space-separated arguments from a string to a command" — is also a waste of time.

Re: GNU Parallel, where have you been all my life?

#177

Post by electroly »

Atotalnoob wrote: post #163
electroly wrote: post #116

Earlier quoted context omitted.

Notably, GNU Parallel did not relicense; it's still GPL. The author wants to have his cake (gain the popularity benefits of being a GPL-licensed GNU tool, be able to carpetbomb Stack Overflow with "use GNU Parallel" answers, etc.) and eat it too (get people to cite or pay him as a condition of using the product). Since this isn't possible (GPL doesn't allow additional restrictions), but the author still really wants…

Shouldn't gnu put a stop to this?

They ensured that the citation request was not actually an additional requirement and has no legal meaning. Beyond that, GNU's interests are better served by retaining GNU Parallel as a GPL-licensed GNU product than by losing it to another organization or another license. I wouldn't expect movement from GNU beyond their existing acknowledgement that the citation request is not a legal requirement and does not modify the GPL. In any event, GNU tends to be hands-off on contributed packages (i.e. the ones that Stallman wasn't involved in writing).

Re: GNU Parallel, where have you been all my life?

#178

Post by zackmorris »

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000.

Contrast that with GPU shaders where one C-style loop operates on buffers separate from system memory, and can't access system services like network sockets or files. GPUs have around 32 or 64 physical cores, so theoretically that many shaders could run simultaneously, although we rarely see that in practice. And we'd need bare-metal drivers to access the GPU cores directly, does anyone know of any?

The closest thing now is Apple's M1 line, but it has specialized NN and GPU cores, so missed out on the potential of true symmetric multiprocessing.

The reason I care about this so much is that with this amount of computing power, kids could run genetic algorithms and other "embarrassingly parallel" code that solves problems about as well as NNs in many cases. Instead we're going to end up with yet another billion dollar bubble that locks us into whatever AI status quo that the tech industry manages to come up with. And everyone seems to love it. It reminds me of the scene in Star Wars III when Padme notes how liberty dies with thunderous applause.

Re: GNU Parallel, where have you been all my life?

#180

Post by JNRowe »

BoppreH wrote: post #107

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

[I'm not recommending this, but maybe… No, no. I'm not sure…]

It isn't even just the newer shells that have solved this, zsh also has a solution out of the box¹. The extensive globbing support in zsh can largely replace `find`, and things like zargs allow you to reuse your common knowledge throughout the shell.

For example, performing your first example with zargs would use regular option separators(`--`), regular expansion(`{1..5}`), and standard shell constructs for the commands to execute.

I'll contrive up an example based around your file counter, but slightly different to show some other functionality.

    f() { fs=($1/*(.)); jo $1=$#fs }
    zargs -P 32 -n1 -- **/*(/) -- f
That should recursively list directories, counting only the files within each, and output² jsonl that can be further mangled within the shell². You could just as easily populate an associative array for further work, or $whatever. Unlike bash, zsh has reasonable behaviour around quoting and whitespace too.

Edit to add: I'm not suggesting zargs is a replacement for parallel, but if you're only using a small subset of its functionality then it may be able to replace that.

¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...

² https://github.com/jpmens/jo

³ https://github.com/stedolan/jq

Post reply on HN