41–50 of 88 posts
Can you explain why you did the naive algorithm here and not any of the fast matrix multiplication ones that trade multiplications for more additions? Just for educational purposes or is there a performance benefit in the technique?
To clarify the title: TFLOP/s is the unit the author goes after, not TFLOP. People in the threads compare CUDA performance on GPUs to WebAssembly performance: please recall that H100 has a theoretical performance of about 1000 TFLOP/s for bfloat16, and even moderately complicated algorithms in typical modern transformer architectures can reach about half of that performance.
For a very deep dive into the subject, this is a great writeup: How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance ( https://siboehm.com/articles/22/CUDA-MMM ) (It's CUDA-specific, so there may be aspects that can't yet be ported to WGPU)
To clarify the title: TFLOP/s is the unit the author goes after, not TFLOP. People in the threads compare CUDA performance on GPUs to WebAssembly performance: please recall that H100 has a theoretical performance of about 1000 TFLOP/s for bfloat16, and even moderately complicated algorithms in typical modern transformer architectures can reach about half of that performance.
H100 can do well over 1500 TFLOPS in fp16.
The datasheet for the H100 SXM seems to indicate that it can only do ~1000 TFLOP/s peak.
For a very deep dive into the subject, this is a great writeup: How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance ( https://siboehm.com/articles/22/CUDA-MMM ) (It's CUDA-specific, so there may be aspects that can't yet be ported to WGPU)
That's a nice tutorial but just to be clear: that is not a deep dive in any sense. It's just the bog standard tricks. It doesn't cover MMA and WMMA, which today is table stakes for fast matmul. Also doesn't cover software pipelining. It's basically a good summary of the basics.
Earlier quoted context omitted.
H100 can do well over 1500 TFLOPS in fp16.
Which H100 and how much over 1500 TFLOP/s? The datasheet for the H100 SXM seems to indicate that it can only do ~1000 TFLOP/s peak.
Great article! For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).
> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.
Earlier quoted context omitted.
75% can't be the best we can do. What would reach 100% or nearly 100%? Handcoded assembly?
The parameters of the matrix multiply, such as the size of the matrices, impose some limits to how close you can get to the peak theoretical performance in a particular GPU. Not all possible matrix multiplies are equally valuable to optimize a priori , so the hardware is designed to perform best on problems that are financially significant, such as modern LLMs. As for handcoded assembly, do you believe that it would…
Why not? We do so for cryptographic primitives and video codecs. And why are you talking about “thousands of kernels”? AI programs only need a small amount of different kernels so it doesn't sound intractable.
Earlier quoted context omitted.
Which H100 and how much over 1500 TFLOP/s? The datasheet for the H100 SXM seems to indicate that it can only do ~1000 TFLOP/s peak.
I just went to Nvidia’s site and downloaded the data sheet: https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor... . It says 1600/1900 in half precision?
Earlier quoted context omitted.
I just went to Nvidia’s site and downloaded the data sheet: https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor... . It says 1600/1900 in half precision?
Read the fine print: "With sparsity". They double the claimed throughput by assuming that half of the FLOPs can be skipped.