Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementing a GPU's programming model on a CPU

litherum.blogspot.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Implementing a GPU's programming model on a CPU

#21

Post by tovej »

mathisfun123 wrote: post #19
tovej wrote: post #17

Earlier quoted context omitted.

Are you even reading the text you cite? That compares SIMD to SIMT. SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does. Most MPI-programs are SPMD, and really, the only conceptual difference between MPI and CUDA is that a thread's id is a point in a grid space that's relevant for scheduling…

> Are you even reading the text you cite? Yes I have read it enough times that I knew exactly where it was in the book. > That compares SIMD to SIMT. Yes and so does the op I responded to. > SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, Thanks but I was already aware. The op was the one that conflated SPMD and SIMD, not me. > SPMD is just a mutiple copies of a single program running on multiple proces…

OP did not conflate them, SPMD and SIMT are the same model, they're just used in different contexts.

Re: Implementing a GPU's programming model on a CPU

#22

Post by tovej »

mathisfun123 wrote: post #19
tovej wrote: post #17

Earlier quoted context omitted.

Are you even reading the text you cite? That compares SIMD to SIMT. SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does. Most MPI-programs are SPMD, and really, the only conceptual difference between MPI and CUDA is that a thread's id is a point in a grid space that's relevant for scheduling…

> Are you even reading the text you cite? Yes I have read it enough times that I knew exactly where it was in the book. > That compares SIMD to SIMT. Yes and so does the op I responded to. > SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, Thanks but I was already aware. The op was the one that conflated SPMD and SIMD, not me. > SPMD is just a mutiple copies of a single program running on multiple proces…

and again, did you even read my reply? You're disagreeing by stating my point back to me. The point about divergence is that IF the threads DONT diverge, you get SIMD scheduling, the GPU can schedule the equivalent of a SIMD OP across the warp.

Obviously they can diverge, because this is SPMD/SIMT

Re: Implementing a GPU's programming model on a CPU

#23

Post by mathisfun123 »

tovej wrote: post #21
mathisfun123 wrote: post #19

Earlier quoted context omitted.

> Are you even reading the text you cite? Yes I have read it enough times that I knew exactly where it was in the book. > That compares SIMD to SIMT. Yes and so does the op I responded to. > SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, Thanks but I was already aware. The op was the one that conflated SPMD and SIMD, not me. > SPMD is just a mutiple copies of a single program running on multiple proces…

OP did not conflate them, SPMD and SIMT are the same model, they're just used in different contexts.

well i guess it's just your word against ... basically any other resource on the matter (in addition to just a common sense understanding) :shrug:

Re: Implementing a GPU's programming model on a CPU

#24

Post by tovej »

mathisfun123 wrote: post #23
tovej wrote: post #21

Earlier quoted context omitted.

OP did not conflate them, SPMD and SIMT are the same model, they're just used in different contexts.

well i guess it's just your word against ... basically any other resource on the matter (in addition to just a common sense understanding) :shrug:

Ok dude, give me an actual argument. Tell me a substantial way that SIMT is different from SPMD

Re: Implementing a GPU's programming model on a CPU

#25

Post by pca006132 »

JonChesterfield wrote: post #18
adrian_b wrote: post #7

This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the CUDA compilers, so the burden for a programmer was greater. As another poster has already mentioned, there exists a compiler for CPUs which has been inspired by CUDA and which has been available for many years: ISPC (Implicit SPMD Program Compiler), at https:/…

SIMT and SIMD are different things. It's fortunate that they have different names. A GPU is a single instruction multiple data machine. That's what the predicated vector operations are. 32 floats at a time, each with a disable bit. Cuda is a single instruction multiple thread language. You write code in terms of one float and branching on booleans, as if it was a CPU, with some awkward intrinsics for accessing the ve…

I guess that a lot of people are uncomfortable thinking about vector instructions, and dealing with masks manually? And for vector instructions you need to align things properly, pad the arrays such that they are of the right size, that people are not used to I guess.

Re: Implementing a GPU's programming model on a CPU

#26

Post by shihab »

JonChesterfield wrote: post #18
adrian_b wrote: post #7

This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the CUDA compilers, so the burden for a programmer was greater. As another poster has already mentioned, there exists a compiler for CPUs which has been inspired by CUDA and which has been available for many years: ISPC (Implicit SPMD Program Compiler), at https:/…

SIMT and SIMD are different things. It's fortunate that they have different names. A GPU is a single instruction multiple data machine. That's what the predicated vector operations are. 32 floats at a time, each with a disable bit. Cuda is a single instruction multiple thread language. You write code in terms of one float and branching on booleans, as if it was a CPU, with some awkward intrinsics for accessing the ve…

I think it boils down to which GPGPU camp you are in, AI or HPC.

AI has relatively simple workflow, less thread divergence, so the SIMT abstractions add very little value. HPC workflow on the other hand is lot more complex. Writing a good simulation program for example, is going to get inhumanly complex with just SIMD.

Re: Implementing a GPU's programming model on a CPU

#27

Post by adrian_b »

hasmanean wrote: post #20
adrian_b wrote: post #7

This so-called GPU programming model has existed many decades before the appearance of the first GPUs, but at that time the compilers were not so good like the CUDA compilers, so the burden for a programmer was greater. As another poster has already mentioned, there exists a compiler for CPUs which has been inspired by CUDA and which has been available for many years: ISPC (Implicit SPMD Program Compiler), at https:/…

Subtle difference. A parallel_for can have asynchronous threads. They can all divergen and run independent instructions (due to if statements etc) SIMT means multiple processors all executing the exact same instruction. One instruction decoder, but like 64 execution pipelines.

The same is true for CUDA/OpenCL kernels. They can include conditionals and they can execute independent instructions.

The difference is that because the software "kernels" (i.e. software threads) may be mapped by the compiler either on hardware threads or on hardware SIMD lanes, and this mapping is not controlled by the programmer, the divergent instructions will cause inefficient (serial) execution when they happen to be executed on SIMD lanes of the same core, so they must be avoided.

This however is only an optimization problem, if the speed would be irrelevant all the kernels could execute divergent instructions.

The reason for the existence of CUDA is to mask for the programmer the existence of the SIMD lanes and to allow programming as if the software threads would map only to hardware threads. Nevertheless, for optimum performance the programmer must be aware that this abstraction is not really true, so the programs should be written with awareness of the limitations introduced by SIMD.

Re: Implementing a GPU's programming model on a CPU

#28

Post by my123 »

adrian_b wrote: post #27
hasmanean wrote: post #20

Earlier quoted context omitted.

Subtle difference. A parallel_for can have asynchronous threads. They can all divergen and run independent instructions (due to if statements etc) SIMT means multiple processors all executing the exact same instruction. One instruction decoder, but like 64 execution pipelines.

The same is true for CUDA/OpenCL kernels. They can include conditionals and they can execute independent instructions. The difference is that because the software "kernels" (i.e. software threads) may be mapped by the compiler either on hardware threads or on hardware SIMD lanes, and this mapping is not controlled by the programmer, the divergent instructions will cause inefficient (serial) execution when they happen…

> Nevertheless, for optimum performance

On conventional SIMT implementations (pre-Volta), the programmer also has to be aware of it to not cause deadlocks in the atomics across different lanes in the same warp.

On NV Volta onwards, each SIMT lane has its own instruction pointer with opportunistic reconvergence when possible.

Re: Implementing a GPU's programming model on a CPU

#29

Post by TechnicolorByte »

> This is in contrast to SIMD, or "single instruction multiple data," where the programmer explicitly uses vector types and operations in their program. The SIMD approach is suited for when you have a single program that has to process a lot of data, whereas SIMT is suited for when you have many programs and each one operates on its own data

This statement is comparing the SIMT model to SIMD. Can anyone explain the last part about SIMT being better for many programs operating on its own data? Are they just saying you can have individual “threads” executing independently (via predication/masks and such)?

Re: Implementing a GPU's programming model on a CPU

#30

Post by raphlinus »

tovej wrote: post #24
mathisfun123 wrote: post #23

Earlier quoted context omitted.

well i guess it's just your word against ... basically any other resource on the matter (in addition to just a common sense understanding) :shrug:

Ok dude, give me an actual argument. Tell me a substantial way that SIMT is different from SPMD

Subgroup operations[1]. In SIMT, they are fairly easily modeled[2] as communication between the different threads running on the same SIMD, and in most cases explicitly expose the predication mask. In fact, subgroupBallot(true) is a common idiom to extract that mask, and also to determine the subgroup size if run in subgroup uniform control flow. Generally they have good performance characteristics.

In SPMD, subgroups aren't easy to model, and would generally be emulated by inter-thread communication if it's important to match the semantics of a source program that includes them. Performance in that case would be terrible.

[1] In Vulkan, OpenGL, OpenCL, and WebGPU they are called "subgroups". In Nvidia including CUDA they are called "warps". In D3D and AMD they are called "waves". In Metal (often running on the same hardware) they are called "simdgroups". This fragmentation of terminology is part of the flavor of working with GPUs.

[2] "Fairly easily" by the standards of GPU infrastructure. In fact, the exact semantics have never been nailed down, and in particular "reconvergence" is poorly defined. Working through this is one of the things blocking subgroups in WebGPU (https://github.com/gpuweb/gpuweb/issues/4306). Even so, they're used commonly in practice, especially for things like fast matrix multiplication where you need to shuffle lots of data into place.

Post reply on HN