Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

271–277 of 277 posts

Re: GNU Parallel, where have you been all my life?

#271

Post by caro11ne »

BoppreH wrote: post #107

It's a nice tool, but it also shows the shortcomings of shell commands. In a proper programming language, we'd have something like parallel [1..5], i => { sleep random()*10+5; possibly_flaky i } // [{"Seq": 4, "Host": ":", "Starttime": 1692491267... And `parallel` would only have to worry about parallelization. Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating…

Do you mean like:

    parallel 'sleep {= $_=rand()*10+5; =} ; possibly_flaky {}' ::: {1..5}
The {= =} escapes to perl, so you have a full programming language available.

Re: GNU Parallel, where have you been all my life?

#272

Post by coliveira »

runeks wrote: post #176
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

If doing even simple things requires looking up documentation, why does it matter whether the shell script is long or short? Spending extra time doing simple things — because you need to Google e.g. "how to pass multiple space-separated arguments from a string to a command" — is also a waste of time.

Because the shell is available everywhere. A programming language needs to be installed first to be of any use. I still understand that there is a need for a simple command processor that is independent of a programming language for simple tasks.

Re: GNU Parallel, where have you been all my life?

#273

Post by runeks »

coliveira wrote: post #272
runeks wrote: post #176

Earlier quoted context omitted.

If doing even simple things requires looking up documentation, why does it matter whether the shell script is long or short? Spending extra time doing simple things — because you need to Google e.g. "how to pass multiple space-separated arguments from a string to a command" — is also a waste of time.

Because the shell is available everywhere. A programming language needs to be installed first to be of any use. I still understand that there is a need for a simple command processor that is independent of a programming language for simple tasks.

> A programming language needs to be installed first to be of any use.

I have that taken care of already

Re: GNU Parallel, where have you been all my life?

#274

Post by flanked-evergl »

DiggyJohnson wrote: post #167
tshaddox wrote: post #144

Earlier quoted context omitted.

“Arbitrary” is probably a better description than “random” here.

I still disagree with arbitrary, just less strongly. Whether his opinion on GPL is relevant, or if it is, how important it is, is up for debate. But I still don’t think it’s “based on a random choice or personal whim, rather than any reason or system”.

If the ride I'm on has the court asking Richard Stallman what he really meant in his heartest of hearts, I want off it.

Re: GNU Parallel, where have you been all my life?

#275

Post by therealcamino »

zackmorris wrote: post #270
imtringued wrote: post #260

Earlier quoted context omitted.

Sorry but we do have computers with 256 cores. I used to have this excuse back when processors only had 4 cores. When you consider that processors lower their turbo boost frequency as you use more cores and there is overhead from synchronization, your 4 core processor may only give you a 2x performance benefit at the expense of your code becoming difficult to reason about (depending on the problem at hand). Nowadays…

That's really awesome, thank you! I agree about the programming models not being parallel by default, and that's one of the things that I specifically rail against in most of my comments. MATLAB/Octave is a good introduction to what parallel programming could be. Also the endless doubling down on large caches, because the multicore design I have in mind would mostly eliminate cache and use that die area for cores and…

Intel Labs experimented with many low power cores vs. fewer, faster high power cores back around 2009-2010.

https://www.zdnet.com/article/experimental-intel-chip-could-...

Programmability is always the biggest issue, and that's not really a chicken-and-egg problem because decades of research have gone into writing compilers and languages for massively parallel machines -- it's just hard, some would say intractable (and local memories tend to make programmability issues worse.) There are niche or embarrasingly-parallel problems that will run great. But it's hard to sell hardware that will solve only some of your problems well. And GPUs have taken over for many of those very regular problems as well.

Re: GNU Parallel, where have you been all my life?

#276

Post by coliveira »

runeks wrote: post #273
coliveira wrote: post #272

Earlier quoted context omitted.

Because the shell is available everywhere. A programming language needs to be installed first to be of any use. I still understand that there is a need for a simple command processor that is independent of a programming language for simple tasks.

> A programming language needs to be installed first to be of any use. I have that taken care of already

This works for you, but many people don't have the luxury of knowing which languages are installed in another system.

Re: GNU Parallel, where have you been all my life?

#277

Post by lysium »

paddim8 wrote: post #248
lysium wrote: post #147

Earlier quoted context omitted.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

Elk is a Shell language with syntax similar to Python. https://elk.strct.net

This looks very interesting! Thank you!
Post reply on HN