Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing a WebGPU Matmul Kernel for 1 TFLOP

zanussbaum.substack.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#61

Post by adev_ »

jsheard wrote: post #19
weinzierl wrote: post #18

Earlier quoted context omitted.

75% can't be the best we can do. What would reach 100% or nearly 100%? Handcoded assembly?

With GPUs it's not uncommon to run out of memory bandwidth before you max out the theoretical FLOPS. They may have a ton of bandwidth but it's never enough. That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.

> That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.

It is not specific to the GPUs: this kind of optimizations are pretty common on CPU too where latency kills you and 200 cycles spent wasted on doing compute can actually be faster than a single cache miss trying to fetch data. This is pretty common for many SIMD algorithms actually.

Memory is currently lagging behind compute on almost every type of modern hardware, and it will very likely become worst, not better.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#62

Post by Const-me »

stephencanon wrote: post #58
Const-me wrote: post #33

Earlier quoted context omitted.

> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.

Probably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.

OP implemented that stuff with WebGPU. In runtime, that weird language compiles into a compute shader which computes that stuff on GPU.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#63

Post by Const-me »

lostmsu wrote: post #51
Const-me wrote: post #4

Couple years ago, I wanted about the same thing in HLSL language, for a Direct3D 11.0 compute shader. Here’s the fastest version I managed to make back then: https://github.com/Const-me/Cgml/blob/master/Mistral/Mistral... As you see, I have implemented 32×32 tiling, using thread groups of 32×8 threads, two groupshared buffers to load tiles of the input matrices, and I accumulate numbers into local variables, 32 / 8 =…

What's the perf like?

Sorry, I have not benchmarked against cuBLAS or Eigen or similar, I did that thing for ML inference.

I have implemented a profiler on top of D3D11_QUERY_TIMESTAMP and D3D11_QUERY_TIMESTAMP_DISJOINT queries, and tweaked the compute shader to minimize the time reported by these queries for my specific use case.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#64

Post by saagarjha »

Const-me wrote: post #62
stephencanon wrote: post #58

Earlier quoted context omitted.

Probably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.

OP implemented that stuff with WebGPU. In runtime, that weird language compiles into a compute shader which computes that stuff on GPU.

I believe the point being made was that this could be done in the CPU faster than was achieved here.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#65

Post by saagarjha »

stephencanon wrote: post #60
jsheard wrote: post #19

Earlier quoted context omitted.

With GPUs it's not uncommon to run out of memory bandwidth before you max out the theoretical FLOPS. They may have a ton of bandwidth but it's never enough. That can lead you to some pretty counter-intuitive optimizations because it's often faster to do more compute work if it means you touch less memory in the process.

For sufficiently large GEMM you should never run out of bandwidth before you max out FLOPS if your blocking is organized correctly, because the arithmetic scales like O(n^3) while the memory access scales like O(n^2).

In theory, yes. In practice you will probably be forced to tile your GEMM and incur the penalty of redundant memory accesses.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#66

Post by saagarjha »

littlestymaar wrote: post #57
david-gpu wrote: post #52

Earlier quoted context omitted.

> AI programs only need a small amount of different kernels That is not the case. What appears like a simple matmul operation actually requires these libraries to select which specific kernel out of the many internally available to execute. If you are curious to learn more, NVidia open sourced a library called Cutlass some years ago. And remember that is only what they are willing to open source.

Is that really different from AV codecs in terms of scale though?

Yes, you can peek under the hood of cuBLAS and notice that it has dozens of kernels for different problem sizes. It’s not generally the case that when you do h264 at a different crf you have a completely different tiling strategy that you have to implement.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#67

Post by shihab »

Const-me wrote: post #33
shihab wrote: post #8

Great article! For context: this WebGPU version achieves ~17% of peak theoretical performance of M2. With CUDA (i.e. CuBLAS), you can reach ~75% of peak performance for same matrix config (without tensor core).

> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.

Yes, that's why I was focusing on percentage of peak hardware performance, not actual flops.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#68

Post by stephencanon »

saagarjha wrote: post #65
stephencanon wrote: post #60

Earlier quoted context omitted.

For sufficiently large GEMM you should never run out of bandwidth before you max out FLOPS if your blocking is organized correctly, because the arithmetic scales like O(n^3) while the memory access scales like O(n^2).

In theory, yes. In practice you will probably be forced to tile your GEMM and incur the penalty of redundant memory accesses.

Sure, but still on each tile, you do O(k^3) compute with O(k^2) memory, and you generally arrange things so that at least one tile is in L1 and at least one other is in L2/LLC (using CPU idioms), so again, you have plenty of bandwidth (typical choices of k are in the ballpark of ~32, and a 32:1 compute to memory ratio is just fine on most hardware, especially if some of those accesses are coming from fast memory)

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#69

Post by Const-me »

saagarjha wrote: post #64
Const-me wrote: post #62

Earlier quoted context omitted.

OP implemented that stuff with WebGPU. In runtime, that weird language compiles into a compute shader which computes that stuff on GPU.

I believe the point being made was that this could be done in the CPU faster than was achieved here.

Yeah, but not on a single core.

In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops.

BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops FP32.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#70

Post by adrian_b »

stephencanon wrote: post #58
Const-me wrote: post #33

Earlier quoted context omitted.

> you can reach ~75% of peak performance for same matrix config Not on the same computer, CUDA doesn’t run on the integrated GPU of the Apple M2 Pro.

Probably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.

Nope, no Apple CPU core has such performance.

That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores.

So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if you have obtained 1 TFLOP/s when running the program on 1 core you will get much more when running it on multiple cores, because all cores of the same type will use the same shared accelerator.

Post reply on HN