Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing a WebGPU Matmul Kernel for 1 TFLOP

zanussbaum.substack.com

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#81

Post by menaerus »

stephencanon wrote: post #75
adrian_b wrote: post #70

Earlier quoted context omitted.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

Right. So, like I said, using one CPU core, you can exceed 1 TFLOP/s, leaving all the other CPU cores and the GPU free for other work.

Top of the line AMD zen5 core can sustain ~80GFLOPS@FP64 and ~160GFLOPS@FP32 using AVX-512, 2x FMA units and ~5Ghz of clock frequency.

This is way way lower than what you claim M2 Pro is capable of and since I'm comparing it against the state-of-the-art datacenter CPU I'm curious how did you get to this number?

M2 Pro core runs at much lower frequency, what it seems to be around ~3.4GHz. And I couldn't find any information about SVE vector widths supported nor number of FMAs.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#82

Post by adrian_b »

stephencanon wrote: post #75
adrian_b wrote: post #70

Earlier quoted context omitted.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

Right. So, like I said, using one CPU core, you can exceed 1 TFLOP/s, leaving all the other CPU cores and the GPU free for other work.

Your initial claim was ambiguous.

It sounded like you claimed that using only one core you already reach 1 TFLOP/s, implying that you could reach more than that by using more cores, which is false.

Now you have clarified that you actually claim that it is good that when using a single core you can reach the maximum throughput of the shared matrix operation accelerator.

This is correct, but there is no essential difference between this and a Zen 5 CPU that reaches this throughput by using only half of the cores, while having the other half of the cores free to do any other tasks.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#83

Post by menaerus »

Const-me wrote: post #69
saagarjha wrote: post #64

Earlier quoted context omitted.

I believe the point being made was that this could be done in the CPU faster than was achieved here.

Yeah, but not on a single core. In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops. BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops…

> Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core.

Isn't it that zen4 doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units?

Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFLOPS number?

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#84

Post by Const-me »

menaerus wrote: post #83
Const-me wrote: post #69

Earlier quoted context omitted.

Yeah, but not on a single core. In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops. BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops…

> Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. Isn't it that zen4 doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units? Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFL…

One FMA counts as two floating-point operations: one multiplication and one addition.

According to uops.info, Zen 4 cores can do two 8-wide FMA instructions per cycle, or one 16-wide FMA per cycle. See VFMADD132PS (YMM, YMM, YMM) and VFMADD132PS (ZMM, ZMM, ZMM) respectively, the throughput column is labelled TP. That’s where 32 FLOP/cycle number comes from.

> doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units

That’s correct, AVX512 doesn’t deliver more FLOPs on that CPU. The throughput of 32-byte FMA and 64-byte FMA is the same, 32 FLOP/cycle for FP32 numbers.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#85

Post by menaerus »

Const-me wrote: post #84
menaerus wrote: post #83

Earlier quoted context omitted.

> Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. Isn't it that zen4 doesn't have "native" support for AVX-512 but "mimics" it through 2x 256-bit FMA units? Because of this, a single AVX-512 instruction will occupy both FMA units and therefore I think that the theoretical limit for a single zen4 core should be half of the 134 GFL…

One FMA counts as two floating-point operations: one multiplication and one addition. According to uops.info, Zen 4 cores can do two 8-wide FMA instructions per cycle, or one 16-wide FMA per cycle. See VFMADD132PS (YMM, YMM, YMM) and VFMADD132PS (ZMM, ZMM, ZMM) respectively, the throughput column is labelled TP. That’s where 32 FLOP/cycle number comes from. > doesn't have "native" support for AVX-512 but "mimics" it…

> One FMA counts as two floating-point operations: one multiplication and one addition.

Right. This is where the discrepancy comes from. I counted FMA as a single FLOP.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#86

Post by stephencanon »

adrian_b wrote: post #82
stephencanon wrote: post #75

Earlier quoted context omitted.

Right. So, like I said, using one CPU core, you can exceed 1 TFLOP/s, leaving all the other CPU cores and the GPU free for other work.

Your initial claim was ambiguous. It sounded like you claimed that using only one core you already reach 1 TFLOP/s, implying that you could reach more than that by using more cores, which is false. Now you have clarified that you actually claim that it is good that when using a single core you can reach the maximum throughput of the shared matrix operation accelerator. This is correct, but there is no essential diffe…

What’s the power draw of however many zen 5 cores you have to tie up to hit, say, 1.5tflop/s on sgemm?

(Also, that’s a M2 number, since that’s what OP was talking about. Someone will presumably post M4 benchmarks for BLAS sometime soon, if they haven’t already.)

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#87

Post by Const-me »

menaerus wrote: post #85
Const-me wrote: post #84

Earlier quoted context omitted.

One FMA counts as two floating-point operations: one multiplication and one addition. According to uops.info, Zen 4 cores can do two 8-wide FMA instructions per cycle, or one 16-wide FMA per cycle. See VFMADD132PS (YMM, YMM, YMM) and VFMADD132PS (ZMM, ZMM, ZMM) respectively, the throughput column is labelled TP. That’s where 32 FLOP/cycle number comes from. > doesn't have "native" support for AVX-512 but "mimics" it…

> One FMA counts as two floating-point operations: one multiplication and one addition. Right. This is where the discrepancy comes from. I counted FMA as a single FLOP.

BTW, it’s the same for GPUs. In DXBC shader byte code, mad instruction does FMA. When reporting theoretical FLOPs, GPU vendors count that as 2 float operations.

For example, I have GeForce 4070 Ti Super in my desktop. The chip has 8448 execution units; nVidia calls them CUDA cores but I don’t like the name, the correct number is 66 cores where each core can do 4 wavefronts of 32 threads each. Anyway, these EUs can do one FP32 FMA each cycle, and the boost clock frequency is 2.61 GHz. Multiplying these two numbers results in 22.04928E+12 cycles*EU/second, and nVidia reports 44E+12 FLOPs peak FP32 performance of the GPU.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#88

Post by forux »

astlouis44 wrote: post #20
jsheard wrote: post #13

Earlier quoted context omitted.

> The smoothness of an iPhone map zoom, on any device. Any device except an iPhone, until Apple finally gets around to shipping WebGPU in Safari. Any year now...

Safari is officially enabling support for WebGPU in iOS 18.2, which is rolling out within the first weeks of December.

I also can confirm that on apple silicon mac webgpu on safari 18.2 works by default, finally.
Post reply on HN