I want a good parallel computer
raphlinus.github.io
I want a good parallel computer
1–10 of 209 posts
Re: I want a good parallel computer
#2> The GPU in your computer is about 10 to 100 times more powerful than the CPU, depending on workload. For real-time graphics rendering and machine learning, you are enjoying that power, and doing those workloads on a CPU is not viable. Why aren’t we exploiting that power for other workloads? What prevents a GPU from being a more general purpose computer?
What other workloads would benefit from a GPU?
Computers are so fast that in practice, many tasks don't need more performance. If a program that runs those tasks is slow, it's because that program's code is particularly bad, and the solution to make the code less bad is simpler than re-writing it for the GPU.
For example, GUIs have been imperceptibly reactive to user input for over 20 years. If an app's GUI feels sluggish, the problem is that the app's actions and rendering aren't on separate coroutines, or the action's coroutine is blocking (maybe it needs to be on a separate thread). But the rendering part of the GUI doesn't need to be on a GPU (any more than it is today, I admit I don't know much about rendering), because responsive GUIs exist today, some even written in scripting languages.
In some cases, parallelizing a task intrinsically makes it slower, because the number of sequential operations required to handle coordination mean there are more forced-sequential operations in total. In other cases, a program spawns 1000+ threads but they only run on 8-16 processors, so the program would be faster if it spawned less threads because it would still use all processors.
I do think GPU programming should be made much simpler, so this work is probably useful, but mainly to ease the implementation of tasks that already use the GPU: real-time graphics and machine learning.
Re: I want a good parallel computer
#3Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them.
Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.
Re: I want a good parallel computer
#4I've always admired the work that the team behind https://www.greenarraychips.com/ does, and the GA144 chip seems like a great parallel computing innovation.
Re: I want a good parallel computer
#5Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.
While acknowledging that it's theoretically possible other approaches might succeed, it seems quite clear the author agrees with you.
Re: I want a good parallel computer
#6It is odd that he talks about Larabee so much, but doesn’t mention the Xeon Phis. (Or is it Xeons Phi?).
> As a general trend, CPU designs are diverging into those optimizing single-core performance (performance cores) and those optimizing power efficiency (efficiency cores), with cores of both types commonly present on the same chip. As E-cores become more prevalent, algorithms designed to exploit parallelism at scale may start winning, incentivizing provision of even larger numbers of increasingly efficient cores, even if underpowered for single-threaded tasks.
I’ve always been slightly annoyed by the concept of E cores, because they are so close to what I want, but not quite there… I want, like, throughput cores. Let’s take E cores, give them their AVX-512 back, and give them higher throughput memory. Maybe try and pull the Phi trick of less OoO capabilities but more threads per core. Eventually the goal should be to come up with an AVX unit so big it kills iGPUs, haha.
Re: I want a good parallel computer
#7Having worked for a company that made a "hundreds of small CPUs on a single chip", I can tell you now that they're all going to fail because the programming model is too weird, and nobody will write software for them. Whatever comes next will be a GPU with extra capabilities, not a totally new architecture. Probably an nVidia GPU.
my take from reading this is more about programming abstractions than any particular hardware instantiation. the part of the Connection Machine that remains interesting is not building machines with CPUS with transistor counts in the hundreds running off a globally synchronous clock, but that there were a whole family of SIMD languages and let you do general purpose programming in parallel. And that those language were still relevant when the architecture changed to a MIMD machine with a bunch of vector units behind each CPU.
Re: I want a good parallel computer
#8> The GPU in your computer is about 10 to 100 times more powerful than the CPU, depending on workload. For real-time graphics rendering and machine learning, you are enjoying that power, and doing those workloads on a CPU is not viable. Why aren’t we exploiting that power for other workloads? What prevents a GPU from being a more general purpose computer? What other workloads would benefit from a GPU? Computers are s…
A big one is video encoding. It seems like GPUs would be ideal for it but in practice limitations in either the hardware or programming model make it hard to efficiently run on GPU shader cores. (GPUs usually include separate fixed-function video engines but these aren't programmable to support future codecs.)
Re: I want a good parallel computer
#9AMD Strix Halo APU is a CPU with very powerful integrated GPU.
It’s faster at AI than an Nvidia RTX4090, because 96GB of the 128GB can be allocated to the GPU memory space. This means it’s doesn’t have the same swapping/memory thrashing that a discrete GPU experiences when processing large models.
16 CPU cores and 40 GPU compute units sounds pretty parallel to me.
Doesn’t that fit the bill?
Re: I want a good parallel computer
#10It is odd that he talks about Larabee so much, but doesn’t mention the Xeon Phis. (Or is it Xeons Phi?). > As a general trend, CPU designs are diverging into those optimizing single-core performance (performance cores) and those optimizing power efficiency (efficiency cores), with cores of both types commonly present on the same chip. As E-cores become more prevalent, algorithms designed to exploit parallelism at sca…
I've always wondered if you could use iGPU compute cores with unified memory as "transparent" E-cores when needed.
Something like OpenCL/CUDA except it works with pthreads/goroutines and other (OS) kernel threading primitives, so code doesn't need to be recompiled for it. Ideally the OS scheduler would know how to split the work, similar to how E-core and P-core scheduling works today.
I don't do HPC professionally, so I assume I'm ignorant to why this isn't possible.