51–60 of 88 posts
Couple years ago, I wanted about the same thing in HLSL language, for a Direct3D 11.0 compute shader. Here’s the fastest version I managed to make back then: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... As you see, I have implemented 32×32 tiling, using thread groups of 32×8 threads, two groupshared buffers to load tiles of the input matrices, and I accumulate numbers into local variables, 32 / 8 =…
Earlier quoted context omitted.
The parameters of the matrix multiply, such as the size of the matrices, impose some limits to how close you can get to the peak theoretical performance in a particular GPU. Not all possible matrix multiplies are equally valuable to optimize a priori , so the hardware is designed to perform best on problems that are financially significant, such as modern LLMs. As for handcoded assembly, do you believe that it would…
> As for handcoded assembly, do you believe that it would be financially sound to hand code and maintain thousands of kernels that way, even if you believed that they would be faster? Why not? We do so for cryptographic primitives and video codecs. And why are you talking about “thousands of kernels”? AI programs only need a small amount of different kernels so it doesn't sound intractable.
That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute.
If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.
Earlier quoted context omitted.
I just went to Nvidia’s site and downloaded the data sheet: https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor... . It says 1600/1900 in half precision?
Read the fine print: "With sparsity". They double the claimed throughput by assuming that half of the FLOPs can be skipped.
I am not an expert in LLM but I don't think you can end up having a significant amount of zeroed weights (~50%) in a converged network so I think it is safe to say that the theoretical throughput for 99% of cases is really ~800 TFLOPS and not ~1600 TFLOPS as advertised.
Great article! For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).
how are you running CUDA on the integrated Apple silicon GPU these days?
Earlier quoted context omitted.
With GPUs it's not uncommon to run out of memory bandwidth before you max out the theoretical FLOPS. They may have a ton of bandwidth but it's never enough. That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.
Shouldn't the roofline inform capacity assessments?
Earlier quoted context omitted.
Read the fine print: "With sparsity". They double the claimed throughput by assuming that half of the FLOPs can be skipped.
Oh, that is really annoying. Thanks for catching that!
Earlier quoted context omitted.
> As for handcoded assembly, do you believe that it would be financially sound to hand code and maintain thousands of kernels that way, even if you believed that they would be faster? Why not? We do so for cryptographic primitives and video codecs. And why are you talking about “thousands of kernels”? AI programs only need a small amount of different kernels so it doesn't sound intractable.
> AI programs only need a small amount of different kernels That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute. If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.
Great article! For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).
> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.
Earlier quoted context omitted.
> AI programs only need a small amount of different kernels That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute. If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.
Is that really different from AV codecs in terms of scale though?
Earlier quoted context omitted.
75% can't be the best we can do. What would reach 100% or nearly 100%? Handcoded assembly?
With GPUs it's not uncommon to run out of memory bandwidth before you max out the theoretical FLOPS. They may have a ton of bandwidth but it's never enough. That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.