Earlier quoted context omitted.
Notably, GNU Parallel did not relicense; it's still GPL. The author wants to have his cake (gain the popularity benefits of being a GPL-licensed GNU tool, be able to carpetbomb Stack Overflow with "use GNU Parallel" answers, etc.) and eat it too (get people to cite or pay him as a condition of using the product). Since this isn't possible (GPL doesn't allow additional restrictions), but the author still really wants…
So if it's GPL, I can remove the citation bits, release it under the GPL as free-parallel, and everything's OK?
GNU Parallel, where have you been all my life?
201–210 of 277 posts
Re: GNU Parallel, where have you been all my life?
#202Re: GNU Parallel, where have you been all my life?
#203I didn't know about this, and reading through the comments, I found out that xargs can also do batching and parallelism (nice!). However, it appears that if you pipe the output of an xargs-parallel command into another utility, it jumbles the output of the multiple subprocesses, whereas GNU parallel does not. I was a little put off by the annoying/scary citation issue mentioned by another commenter, so I am not sure…
You can probably do something that creates as many FIFOs as you have parallelism and just be careful about emitting whole records like https://github.com/c-blake/bu/blob/main/doc/funnel.md . That one's Nim, but the meat is only like 50 lines and easily ported to C like your progress tool. ( EDIT: and it will also probably be drastically lower overhead than `parallel` which has over 70X worse time overhead and 10X the…
Thanks for the suggestion!
Re: GNU Parallel, where have you been all my life?
#204You guys know that in bash you can use `&` to pass a foreground terminal process to the background and then use `wait` to wait for all the session's background process to end, right?
Strong "Dropbox is just rsync it'll never sell" vibes.
Re: GNU Parallel, where have you been all my life?
#205Earlier quoted context omitted.
Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?
You've some hesitation with Perl, but if you stick at it, you'll find what you seek. It feels very 'unixy' and can achieve much the same as shell while being more consistent in its syntax. Its portability means it will work the same across environments. Plus the newest editions have niceties like modern classes and try/catch as inbuilt language features. Sharing this because its the route I went, anything I'd have wr…
The tooling around Perl has also gotten better over the last decade or so while also allowing you to pack everything to run on even ancient machines running old Perl 5.
Re: GNU Parallel, where have you been all my life?
#206Earlier quoted context omitted.
You can probably do something that creates as many FIFOs as you have parallelism and just be careful about emitting whole records like https://github.com/c-blake/bu/blob/main/doc/funnel.md . That one's Nim, but the meat is only like 50 lines and easily ported to C like your progress tool. ( EDIT: and it will also probably be drastically lower overhead than `parallel` which has over 70X worse time overhead and 10X the…
Thanks for the suggestion!
Also, the last time I tried, to do similar with FIFOs (no /tmp | whatever storage like other e.g.s here https://news.ycombinator.com/item?id=37211687), GNU parallel needed some - for me - specially compiled Perl interpreter with threads enabled to use its `parcat` program which is also probably slow. Besides the nagware insanity, `parallel` seems just not a very compelling tool in either machine|human overheads unless -- maybe -- you already know Perl (which I always found a supremely forgettable language).
Re: GNU Parallel, where have you been all my life?
#207Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…
1000 cores?? I don't have 100 cores! What do you even need 10 cores for? Well, here's 4 cores. Give 2 to your brother. Don't go wasting all those hyper threads all at once! Intel ca. 2010, probably
Also Intel: ECC memory support? In this economy?
Re: GNU Parallel, where have you been all my life?
#208Earlier quoted context omitted.
> IIRC the citation notice was cleared by Stallman as GPL compatible Do you have a source for this? Im confused by this, as the GPL section 7 is pretty clear that additional restrictions are effectively void. I suppose it’s technically not contrary to the GPL to idly state those restrictions, but it is contrary to the GPL to expect them to do anything. If the author is deliberately including an impotent clause in the…
Here’s one source, there are several other places Ole has talked about it: https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita... I think the confusing issue here is that the notice is not a license requirement, it does not add additional licensing restrictions. It’s an honor-system agreement between the user and Ole, and does not involve the GPL. It does seem to be walking a very fine line, and it’s easy fo…
Yes, to me it looks like he’s adding an official license-like note, but then declaring that he’s still GPL compliant because although his note is easily confused with a license it’s not actually a license. He then gets cranky if people remove his not-a-license note or don’t act like it’s a license. Feels very much to me like he’d be better served with something other than the GPL if he doesn’t want people using his software in GPL-permitted ways.
Re: GNU Parallel, where have you been all my life?
#209Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…
Typical GPUs are easily 6000+ shaders (aka kinda-sorta like cores) on the more expensive end.
At least, 6000+ 32-bit multiplies per clock tick on ~2GHz+ clocks. Even cheap GPUs easily are 2000+ shaders.
> GPUs have around 32 or 64 physical cores
NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism.
As far as grunt-work goes, a "multiplier unit" (literally A x B) is perhaps the most accurate count to compare CPU cores vs GPU "cores", because the concept of CPU-core vs GPU WGP / SM is too weird and different to directly compare.
Split up that WGP / SM into individual multipliers... and also split up the ~3 64-bit multipliers or ~48 CPU SIMD multipliers per core (3x 512-bit on Intel AVX512 cores), and its perhaps a more fair comparison point.
---------
Back 20 years ago, you'd only have 1x multiplier on a CPU core like a Pentium 4, maybe as many as 4x with the 128-bit SSE instructions.
But today, even 1x core from Intel (3x 512-bit SIMD) or 1x core from AMD (4x 256-bit SIMD) has many, many, many more parallel elements compared to a 2004-era CPU core.
Re: GNU Parallel, where have you been all my life?
#210Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…
Arguing about where we should be based on a projection of an empirical exponential curve seems pretty irrational. Nothing in reality is exponential forever.