I want a good parallel computer • Hacker News Board

Re: I want a good parallel computer

#171

Post by wmf » Sun, Mar 23, 2025, 4:00 AM UTC

aleph_minus_one wrote: post #142

deviantbit wrote: post #49

"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a re…

> What amazes me more is the comment section—full of people waxing nostalgic for architectures they clearly never had to ship stable software on. Isn't it much more plausible that the people who love to play with exotic (or also retro), complicated architectures (with in this case high performance opportunities) are different people than those who love to "set up or work in an assembly line for shipping stable softwa…

The problem is that the exotic complexity enthusiasts cluster in places like HN and sometimes they overwhelm the voices of reason.

Re: I want a good parallel computer

#172

Post by dragontamer » Sun, Mar 23, 2025, 4:09 AM UTC

raphlinus wrote: post #103

dragontamer wrote: post #95

There's a lot here that seems to misunderstand GPUs and SIMD. Note that raytracing is a very dynamic problem, where the GPU isn't sure if a ray hits a geometry or if it misses. When it hits, the ray needs to bounce, possibly multiple times. Various implementations of raytracing, recursion, dynamic parallelism or whatever. Its all there. Now the software / compilers aren't ready (outside of specialized situations like…

The problems I'm having are very different than those for raytracing. Sure, it's dynamic, but at a fine granularity, so the problems you run into are divergence, and often also wanting function pointers, which don't work well in a SIMT model, By contrast, the way I'm doing 2D there's basically no divergence (monoids are cool that way) but there is a need to schedule dynamically at a coarser (workgroup) level. But the…

> But the biggest problem I'm having is management of buffer space for intermediate objects. That's not relevant to the core of raytracing because you're fundamentally just accumulating an integral, then writing out the answer for a single pixel at the end.

True Allocation just seems to be a "forced sequential" operation. A "stop the world, figure out what RAM is available" kind of thing.

If you can work with pre-allocated buffers, then GPUs work by reading from lists (consume operations), and then outputting to lists (append operations). Which can be done with gather / scatter, or more precisely stream-expansion and stream-compaction in a grossly parallel manner.

---------

If that's not enough "memory management" for you, then yeah, CPU is the better device to work with. At which point I again point back to the 192-core EPYC Zen5c example, we have grossly parallel CPUs today if you need them. Just a few clicks away to rent from cloud providers like Amazon or Azure.

GPUs are good at certain things (and I consider the pinnacle of "Connection Machine" style programming. Just today's GPUs are far more parallel, far easier to program and far faster than the old 1980s stuff).

Some problems cannot be split up (ex: web requests are so unique I cannot imagine they'd ever be programmed into a GPU due to their divergence). However CPUs still exist for that.

Re: I want a good parallel computer

#173

Post by dragontamer » Sun, Mar 23, 2025, 4:13 AM UTC

jms55 wrote: post #159

raphlinus wrote: post #103

Earlier quoted context omitted.

The problems I'm having are very different than those for raytracing. Sure, it's dynamic, but at a fine granularity, so the problems you run into are divergence, and often also wanting function pointers, which don't work well in a SIMT model, By contrast, the way I'm doing 2D there's basically no divergence (monoids are cool that way) but there is a need to schedule dynamically at a coarser (workgroup) level. But the…

Agreed, there are two different problems being described here. 1. Divergence of threads within a workgroup/SM/whatever 2. Dynamically scheduling new workloads (i.e. dispatches, draws, etc) in response to the output of a previous workload Raytracing is problem #1 (and has it's own solutions, like shader execution reodering), while Raph is talking about problem #2.

> Raytracing is problem #1 (and has it's own solutions, like shader execution reodering)

The "solution" to Raytracing (ignoring hardware acceleration like shader reordering), is stream compaction and stream expansion.

    if (ray hit){ 
        push(hits_array, currentRay); 
    } else { 
        push (miss_array, currentRay); 
    }

If you are willing to have lots of loops inside of a shader (not always possible due to Windows's 2 second maximum), you can while(hits_array is not empty) kind of code, allowing your 1024-wavegroup to keep recursively calling all of the hits and efficiently processing all of the rays recursively.

--------

The important tidbit is that this technique generalizes. If you have 5 functions that need to be "called" after your current processing, then it becomes:

    if (func1 needs to be called next){ 
        push(func1, dataToContinue);
    } else if (func2 needs to be called next){ 
        push(func2, dataToContinue);
    } else if (func3 needs to be called next){ 
        push(func3, dataToContinue);
    } else if (func4 needs to be called next){ 
        push(func4, dataToContinue);
    } else if (func5 needs to be called next){ 
        push(func5, dataToContinue);
    }

Now of course we can't grow "too far", GPUs can't handle divergence very well. But for "small" numbers of next-arrays and "small" amounts of divergence (ie: I'm assuming that func1 is the most common here, like 80%+ so that the buffers remain full), then this technique works.

If you have more divergence than that, then you need to think more carefully about how to continue. Maybe GPUs are a bad fit (ex: any HTTP server code will be awful on GPUs) and you're forced to use a CPU.

Re: I want a good parallel computer

#174

Post by fulafel » Sun, Mar 23, 2025, 6:36 AM UTC

tliltocatl wrote: post #72

grg0 wrote: post #19

The issue is that programming a discrete GPU feels like programming a printer over a COM port, just with higher bandwidths. It's an entirely moronic programming model to be using in 2025. - You need to compile shader source/bytecode at runtime; you can't just "run" a program. - On NUMA/discrete, the GPU cannot just manipulate the data structures the CPU already has; gotta copy the whole thing over. And you better des…

Larrabee was something like that, didn't took off. IMHO, the real issue is cache coherence. GPUs are spared from doing a lot of extra work by relaxing coherence guarantees quite a bit. Regarding the vendor situation - that's basically how most of computing hardware is, save for the PC platform. And this exception is due to Microsoft successfully commoditizing their complements (which caused quite some woe on the soft…

In the field usually nothing takes off on the first attempt, so this is just a reason to ask "what's different this time" on the following attempts.

Re: I want a good parallel computer

#175

Post by eternityforest » Sun, Mar 23, 2025, 8:57 AM UTC

I wonder if CDN server applications could use something like this, if every core had a hardware TCP/TLS stack and there was a built-in IP router to balance the load, or something like that.

Re: I want a good parallel computer

#176

Post by [deleted] » Sun, Mar 23, 2025, 8:59 AM UTC

dist-epoch wrote: post #18

wmf wrote: post #8

Earlier quoted context omitted.

A big one is video encoding. It seems like GPUs would be ideal for it but in practice limitations in either the hardware or programming model make it hard to efficiently run on GPU shader cores. (GPUs usually include separate fixed-function video engines but these aren't programmable to support future codecs.)

Video encoding is done with fixed-function for power efficiency. A new popular codec like H26x codec appears every 5-10 years, there is no real need to support future ones.

[deleted]

Re: I want a good parallel computer

#177

Post by muziq » Sun, Mar 23, 2025, 9:10 AM UTC

I was always fascinated by the prospects of the 1024-core Epiphany-V from Parallella.. https://parallella.org/2016/10/05/epiphany-v-a-1024-core-64-... But it seems whatever the DARPA connection was has led to it not being for scruffs like me and is likely powering god knows what military systems..

Re: I want a good parallel computer

#178

Post by hulitu » Sun, Mar 23, 2025, 10:16 AM UTC

Diggsey wrote: post #136

deviantbit wrote: post #49

"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a re…

> I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. So the designers of the Cell processor made some mistakes and therefore the entire concept is bunk? Because you've seen a concept done badly, you can't imagine it done well? To be clear, I'm not criticising those designers, they probably did a g…

> and we've figured out how to securely isolate "users" of the same GPU from each other

That's the problem, isn't it.

I don't want my programs to act independently, they need to exchange data with each other (copy-paste, drag and drop). Also i cannot do many things in parralel. Some thing must be done sequencially.

Re: I want a good parallel computer

#179

Post by hulitu » Sun, Mar 23, 2025, 10:17 AM UTC

Yoric wrote: post #107

deviantbit wrote: post #49

"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a re…

Don't worry, with LLMs, we're moving away from anything that remotely looks like "stable software" :) Also, yeah, I recall the dreaded days of cooperative multitasking between apps. Moving from Windows 3.x to Linux was a revelation.

With LLM's it is just more visible. When the age of "updates" begun, the age of stable software died.

Re: I want a good parallel computer

#180

Post by mabster » Sun, Mar 23, 2025, 11:07 AM UTC

deviantbit wrote: post #49

"I believe there are two main things holding it back." He really science’d the heck out of that one. I’m getting tired of seeing opinions dressed up as insight—especially when they’re this detached from how real systems actually work. I worked on the Cell processor and I can tell you it was a nightmare. It demanded an unrealistic amount of micromanagement and gave developers rope to hang themselves with. There’s a re…

I loved and really miss the cell. It did take quite a bit of work to shuffle things in and out of the SPUs correctly (so yeah, it took much longer to write code and greater care), but it really churned through data.

We had a generic job mechanism with the same restrictions on all platforms. This usually meant if it ran at all on Cell it would run great on PC because the data would generally be cache friendly. But it was tough getting the PowerPC to perform.

I understand why the PS4 was basically a PC after that - because it's easier. But I wish there was still SPUs off the side to take advantage of. Happy to have it off die like GPUs are.