I want a good parallel computer

"The birth and death of JavaScript"

The problems I'm having are very different than those for raytracing. Sure, it's dynamic, but at a fine granularity, so the problems you run into are divergence, and often also wanting function pointers, which don't work well in a SIMT model, By contrast, the way I'm doing 2D there's basically no divergence (monoids are cool that way) but there is a need to schedule dynamically at a coarser (workgroup) level.

But the biggest problem I'm having is management of buffer space for intermediate objects. That's not relevant to the core of raytracing because you're fundamentally just accumulating an integral, then writing out the answer for a single pixel at the end.

The problem with the GPU raytracing work is that they built hardware and driver support for the specific problem, rather than more general primitives on which you could build not only raytracing but other applications. The same story goes for video encoding. Continuing that direction leads to unmanageable complexity.

Of course today's machines are better, they have orders of magnitude more transistors, and crystallize a ton of knowledge on how to build efficient, powerful machines. But from a design aesthetic perspective, they're becoming junkheaps of special-case logic. I do think there's something we can learn from the paths not taken, even if, quite obviously, it doesn't make sense to simply duplicate older designs.

I'm going to call this out. The entire post obviously has bucket loads if aggression which can be taken as just communication style, but the last line was just uncalled for.

I have seen you make high quality responses to crazy posts.

Do better.

Don't worry, with LLMs, we're moving away from anything that remotely looks like "stable software" :)

Also, yeah, I recall the dreaded days of cooperative multitasking between apps. Moving from Windows 3.x to Linux was a revelation.

What you say has nothing to do with CPU vs. GPU, or with CUDA, which is basically equivalent with the older OpenMP.

When you have a set of concurrent threads, each thread may run a different program. There are many applications where this is necessary, but such applications are hard to scale to very high levels of concurrency, because each thread must be handled individually by the programmer.

Another case is when all the threads run the same program, but on different data. This is equivalent with a concurrent execution of a "for" loop, which is always possible when the iterations are independent.

The execution of such a set of threads that execute the same program has been named "parallel DO instruction" by Melvin E. Conway in 1963, "array of processes" by C. A. R. Hoare in 1978, "replicated parallel" in the Occam programming language in 1985, SPMD around the same time, "PARALLEL DO" in the OpenMP Fortran language extension in 1997, "parallel for" in the OpenMP C/C++ language extension in 1998, and "kernel execution" in CUDA, which has also introduced the superfluous acronym SIMT to describe it.

When a problem can be solved by a set of concurrent threads that run the same program, then it is much simpler to scale the parallelism to extremely high levels and the parallel execution can usually be scheduled by a compiler or by a hardware controller without the programmer having to be concerned with the details.

There is no inherent difficulty in making a compiler that provides exactly the same programming model as CUDA, but which creates a program for a CPU, not for a GPU. Such compilers exist, e.g. ispc, which is mentioned in the parent article.

The difference between GPUs and CPUs is that the former appear to have some extra hardware support for what you describe as "distributing that to the compute cores and managing the data flows between them", but nobody is able to tell exactly what is done by this extra hardware support and whether it really matters, because it is a part of the GPUs that has never been documented publicly by the GPU vendors.

From the point of view of the programmer, this possible hardware advantage of the GPUs does not really matter, because there are plenty of programming language extensions for parallelism and libraries that can take care of the details of thread spawning and work distribution over SIMD lanes, regardless if the target is a CPU or a GPU.

Whenever you write a program equivalent with a "parallel for", which is the same as writing for CUDA, you do not manage individual threads, because what you write, the "kernel" in CUDA lingo, can be executed by thousands of threads, also on a CPU, not only on a GPU. A desktop CPU like Ryzen 9 9950X has the same product of threads by SIMD lanes like a big integrated GPU (obviously, discrete GPUs can be many times bigger).