Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GNU Parallel, where have you been all my life?

alexplescan.com

Post reply on HN
Print view View original

261–270 of 277 posts

Re: GNU Parallel, where have you been all my life?

#261

Post by imtringued »

dragontamer wrote: post #209
zackmorris wrote: post #178

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

Typical GPUs are easily 6000+ shaders (aka kinda-sorta like cores) on the more expensive end. At least, 6000+ 32-bit multiplies per clock tick on ~2GHz+ clocks. Even cheap GPUs easily are 2000+ shaders. > GPUs have around 32 or 64 physical cores NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism. As far as grunt-work goes, a "m…

>NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism.

They aren't weird things. They are the equivalent of CPU cores. By your logic CPU cores aren't CPU cores, "they are... weird things" because of SMT.

Re: GNU Parallel, where have you been all my life?

#262

Post by lysium »

cb321 wrote: post #245
lysium wrote: post #242

Earlier quoted context omitted.

Thanks for pointing me to nim, it looks promising. I‘ll try to use https://nim-lang.org/docs/osproc.html to pipe programs. My use case is approx. like this: I can get 80% what I want with ls … | sed … | grep -v … but then it gets complicated in the script and I’d like to replace the sed or grep part with some program.

This sounds like a job for what standard C calls "popen". You can do import posix; for line in popen("ls", "r").lines: echo line in Nim, though you obviously need to replace `echo line` with other desired processing and learn how to do that. You might also want to consider `rp` which is a program generator-compiler-runner along the lines of `awk` but with all the code just Nim snippets interpolated into a program tem…

Thank you! Popen looks like what I was looking for!

Re: GNU Parallel, where have you been all my life?

#263

Post by imtringued »

shakow wrote: post #189
rlpb wrote: post #128

Earlier quoted context omitted.

> and the citation notice is a one-time thing you can silence permanently This doesn't scale. Imagine if all the software you used nagged you and had their own individual methods to silence them. I don't think this would be reasonable. What makes this particular software so special?

> This doesn't scale. --will-cite

There is a difference between "will cite" and being in a situation that warrants citing.

Re: GNU Parallel, where have you been all my life?

#264

Post by fho »

lysium wrote: post #147
coliveira wrote: post #127

Earlier quoted context omitted.

What you mention is the main reason why shell script is not a decent language to write long programs. It is full of inconsistencies, and since it depends on other commands, you have to learn the quirks of each command you use. Moreover, good luck if you need to debug this. Shell should only be used for small scripts that are easy to debug.

Do you recommend any good alternative when your shell program gets too large? Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?

Unpopular opinion, but I used Haskell "scripts" with relative success for a while. Stack has a nice script interpreter mode that is runnable in the familiar #! way.

Even allows to add dependencies and if necessary compile the script on the fly.

Re: GNU Parallel, where have you been all my life?

#265

Post by cb321 »

lysium wrote: post #262
cb321 wrote: post #245

Earlier quoted context omitted.

This sounds like a job for what standard C calls "popen". You can do import posix; for line in popen("ls", "r").lines: echo line in Nim, though you obviously need to replace `echo line` with other desired processing and learn how to do that. You might also want to consider `rp` which is a program generator-compiler-runner along the lines of `awk` but with all the code just Nim snippets interpolated into a program tem…

Thank you! Popen looks like what I was looking for!

Sure. No problem.

While it may not be in any ANSI/ISO spec for C, even Windows has popen these days. There are also some tiny Nim popenr/popenw wrappers in https://github.com/c-blake/cligen/blob/master/cligen/osUt.ni... covering the Windows case.

Depending upon how balanced work is on either side of the pipe, you usually can even get parallel speed-up on multicore with almost no work. For example, there is no need to use quote-escaped CSV parsing libraries when you just read from a popen()d translator program producing an easier format: https://github.com/c-blake/nio/blob/main/utils/c2tsv.nim

Re: GNU Parallel, where have you been all my life?

#267

Post by dragontamer »

imtringued wrote: post #261
dragontamer wrote: post #209

Earlier quoted context omitted.

Typical GPUs are easily 6000+ shaders (aka kinda-sorta like cores) on the more expensive end. At least, 6000+ 32-bit multiplies per clock tick on ~2GHz+ clocks. Even cheap GPUs easily are 2000+ shaders. > GPUs have around 32 or 64 physical cores NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism. As far as grunt-work goes, a "m…

>NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism. They aren't weird things. They are the equivalent of CPU cores. By your logic CPU cores aren't CPU cores, "they are... weird things" because of SMT.

There is more weirdness here than just SMT.

The full crossbar, allowing each shader to individually issue a fetch from memory. The shared memory space is not like cache but instead is a shader-to-shader communication scratchpad.

Atomics support, coalescing atomics together.

-------

I mean hell: what is a core? Do remember that on SMs, every single shader (not SM) has its own instruction pointer.

Is the shader a core? No, not really. But SMs aren't a core either.

I wouldn't compare GPU and CPU architecture at all. They're just different. What I did above, breaking both down into individual multipliers then counting them seems like the best way forward, especially as we remain multiplier bound in practice.

Re: GNU Parallel, where have you been all my life?

#268

Post by wiredfool »

c-hendricks wrote: post #237
wiredfool wrote: post #40

Earlier quoted context omitted.

Xargs is a nearly drop in replacement and probably already installed by default in most distros. You may need the -n 1 (one file per) and -P to parallelize. xargs -n 1 -P 8

find + xargs has become my go-to "process files in parallel". Tho now I'm wondering if I should be using `-n` instead of `-L` #!/usr/bin/env bash set -e main() { if [ "$1" = "handle-file" ]; then shift handle-file "$@" else find . \ -type f \ -not -path '*/optimized/*' \ -print0 \ | xargs \ -0 \ -L 1 \ -P 8 \ -I {} \ bash -c "cd \"$PWD\" && \"$0\" handle-file \"{}\"" fi } handle-file() { echo "handle-file $1 ..." } m…

Huh, I hadn't seen the -L before. Looks pretty similar.

Re: GNU Parallel, where have you been all my life?

#269

Post by zackmorris »

JonChesterfield wrote: post #191
zackmorris wrote: post #178

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

This exists now. Some AI accelerators are a grid of independent compute units with their own memory, message passing between them. Graphcore's IPU is an instance. An AMD GPU is a grid of independent compute units on a memory hierarchy. At the fine grain, it's a scalar integer unit (branches, arithmetic) and a predicated vector unit, with an instruction pointer. Ballpark of 80 of those can be on a given compute unit a…

K if I can transpile C/C++, Rust or TypeScript to that and have full access to memory, threads, system APIs, network sockets, etc, then that would work for the use cases I have in mind. Running MIMD processes on SIMD hardware is something I'm definitely interested in.

If there's no straightforward way to do that, then I'm afraid that hardware represents a huge investment in the wrong direction.

Because a GPU can be built from the general-purpose multicore CPU I'm talking about. But a CPU can't be built from a GPU.

What I'm getting at is that if I have to "drop down" to an orthodox way of solving problems, rather than being able to solve them in the freeform way that my instincts leads me, then I will always be stifled.

Re: GNU Parallel, where have you been all my life?

#270

Post by zackmorris »

imtringued wrote: post #260
zackmorris wrote: post #178

Since nobody asked, I'm reiterating my position that computers to effectively utilize parallel functionality simply aren't available today. I've always wanted a computer with at least 256 cores and local content-addressable memories beside each core to send data where it's needed. By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000. Contra…

Sorry but we do have computers with 256 cores. I used to have this excuse back when processors only had 4 cores. When you consider that processors lower their turbo boost frequency as you use more cores and there is overhead from synchronization, your 4 core processor may only give you a 2x performance benefit at the expense of your code becoming difficult to reason about (depending on the problem at hand). Nowadays…

That's really awesome, thank you!

I agree about the programming models not being parallel by default, and that's one of the things that I specifically rail against in most of my comments. MATLAB/Octave is a good introduction to what parallel programming could be. Also the endless doubling down on large caches, because the multicore design I have in mind would mostly eliminate cache and use that die area for cores and local memories.

I think we're slightly talking past each other here though. The CPU I want to build would have around 10-256 cores on 90s tech. So the same transistors holding 1 Pentium Pro would allow for 1-2 orders of magnitude more MIPS or RISC-V cores and local memories. The design is so simple that I think that's why it was missed by the big fabs.

Today there's little demand for 1000+ cores, but that's partly because nobody can see what they could do. But we can't design the thing, because the status quo has us all working pedal to the metal in first gear to make rent. It's a chicken and egg problem that has a lower likelihood of being solved as time goes on. Which is why I think we're on the wrong timeline, because if the system worked then actual innovation would become more accessible over time.

Post reply on HN