Optimizing a WebGPU Matmul Kernel for 1 TFLOP
zanussbaum.substack.com
Optimizing a WebGPU Matmul Kernel for 1 TFLOP
1–10 of 88 posts
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#2WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#3WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
great question, to me webGPU sits a hair high level than CUDA or Vulkan. so you don't have the exact same level of control but can get to 80% performance of it without having to write different kernels specific to the hardware
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#4Couple years ago, I wanted about the same thing in HLSL language, for a Direct3D 11.0 compute shader. Here’s the fastest version I managed to make back then: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral...
As you see, I have implemented 32×32 tiling, using thread groups of 32×8 threads, two groupshared buffers to load tiles of the input matrices, and I accumulate numbers into local variables, 32 / 8 = 4 accumulators per thread.
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#5WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#6WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications
you're definitely right, 80% was a bit of an overestimation, especially with respect to CUDA
it would be cool to see if there's some way to get better access to those lower-level primitives but would be surprised
it does seem like subgroup support are a step in the right direction though!
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#7WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications
> don't have support for hardware specific memory
I have no experience with WebGPU but if you mean group shared memory, I think the support is available. See the demo: https://compute.toys/view/25
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#8Great article!
For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#9Earlier quoted context omitted.
WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications
> don't have support for hardware specific memory I have no experience with WebGPU but if you mean group shared memory, I think the support is available. See the demo: https://compute.toys/view/25
i tried using workgroup shared memory and found it slower than just recomputing everything in each thread although i may have been doing something dumb
i'm excited to try subgroups though: https://developer.chrome.com/blog/new-in-webgpu-128#experime...
Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP
#10WebGPU doesn't seem to talk about bank conflict, hiding some hardware details that might be necessary to write the best kernel. will it be able to match the perf of Cuda on the same hardware?
WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications
I've heard the WebGPU workgroup wants to close the gap on tensor core support.