Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizing a WebGPU Matmul Kernel for 1 TFLOP

zanussbaum.substack.com

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#71

Post by Const-me »

adrian_b wrote: post #70
stephencanon wrote: post #58

Earlier quoted context omitted.

Probably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

One interesting thing about these newfangled matrix/AI/ML accelerators that’s very rarely mentioned on the internets, they only deliver that many TFLOP because they operate in very low precision.

nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16.

None of them supports FP32 let alone FP64 input numbers, which makes them completely useless for traditional GEMM stuff.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#72

Post by wtallis »

Const-me wrote: post #71
adrian_b wrote: post #70

Earlier quoted context omitted.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

One interesting thing about these newfangled matrix/AI/ML accelerators that’s very rarely mentioned on the internets, they only deliver that many TFLOP because they operate in very low precision. nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16. None of them supports FP32 let alone FP64 inp…

https://github.com/corsix/amx indicates that Apple's AMX supports up to 64-bit FP, but I don't see any performance metrics. They also have the ANE, which is the low-precision ML-focused accelerator.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#73

Post by Const-me »

wtallis wrote: post #72
Const-me wrote: post #71

Earlier quoted context omitted.

One interesting thing about these newfangled matrix/AI/ML accelerators that’s very rarely mentioned on the internets, they only deliver that many TFLOP because they operate in very low precision. nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16. None of them supports FP32 let alone FP64 inp…

https://github.com/corsix/amx indicates that Apple's AMX supports up to 64-bit FP, but I don't see any performance metrics. They also have the ANE, which is the low-precision ML-focused accelerator.

I wasn’t aware there’re two completely different things from different companies both called AMX. I assumed that AMX: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions

The Apple’s version is indeed interesting. I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#74

Post by wtallis »

Const-me wrote: post #73
wtallis wrote: post #72

Earlier quoted context omitted.

https://github.com/corsix/amx indicates that Apple's AMX supports up to 64-bit FP, but I don't see any performance metrics. They also have the ANE, which is the low-precision ML-focused accelerator.

I wasn’t aware there’re two completely different things from different companies both called AMX. I assumed that AMX: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions The Apple’s version is indeed interesting. I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?

> I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?

Using the Accelerate framework (which includes Apple's BLAS) is the only supported way for programmers to access the AMX. Reverse engineering the instruction set to access it directly is discouraged, because it's not a documented stable interface.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#75

Post by stephencanon »

adrian_b wrote: post #70
stephencanon wrote: post #58

Earlier quoted context omitted.

Probably more relevant here is that a single CPU core on that computer exceeds 1 tflop/s on gemm with plenty of margin using a single lib call, and leaves the rest of the CPU cores and all of the GPU free to do other work.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

Right. So, like I said, using one CPU core, you can exceed 1 TFLOP/s, leaving all the other CPU cores and the GPU free for other work.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#76

Post by stephencanon »

Const-me wrote: post #71
adrian_b wrote: post #70

Earlier quoted context omitted.

Nope, no Apple CPU core has such performance. That single lib call must have used the AMX accelerator, which is separate from the cores and shared by a group of cores. So that AMX accelerator performance may be greater than of all CPU cores together. AFAIK, some Apple CPUs have one AMX accelerator for the big cores and another AMX accelerator for the smaller cores, but in any case there is no chance to hope that if y…

One interesting thing about these newfangled matrix/AI/ML accelerators that’s very rarely mentioned on the internets, they only deliver that many TFLOP because they operate in very low precision. nVidia tensor cores support int8, couple versions of FP16 (BF16 and the standard IEEE one) and FP19 which they call TensorFloat-32. I think Intel AMX only supports int8 and BF16. None of them supports FP32 let alone FP64 inp…

Apple's matrix unit supports FP16, 32, and 64 sources and accumulators.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#77

Post by [deleted] »

pama wrote: post #29

To clarify the title: TFLOP/s is the unit the author goes after, not TFLOP. People in the threads compare CUDA performance on GPUs to WebAssembly performance: please recall that H100 has a theoretical performance of about 1000 TFLOP/s for bfloat16, and even moderately complicated algorithms in typical modern transformer architectures can reach about half of that performance.

[deleted]

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#78

Post by sgerenser »

Const-me wrote: post #73
wtallis wrote: post #72

Earlier quoted context omitted.

https://github.com/corsix/amx indicates that Apple's AMX supports up to 64-bit FP, but I don't see any performance metrics. They also have the ANE, which is the low-precision ML-focused accelerator.

I wasn’t aware there’re two completely different things from different companies both called AMX. I assumed that AMX: https://en.wikipedia.org/wiki/Advanced_Matrix_Extensions The Apple’s version is indeed interesting. I wonder why haven’t Apple exposed it to programmers, or implemented a BLAS library on top of that thing?

It’s because Apple’s AMX was an early, in-house built version of what was eventually released by Arm as SME. Apple adopted SME in the M4 and dropped AMX, but as long as you were using their Accelerate framework instead of directly writing AMX code (which they told people not to do), you wouldn’t notice.

Now that they’re using “standard” SME, it shouldn’t be a problem to write SME assembly opcodes directly, although I suspect Apple themselves is still probably sparse on the documentation. I’m not aware if there’s any way to use intrinsics or something slightly higher level than inline-ASM, but lower level than the Accelerate framework.

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#79

Post by saagarjha »

stephencanon wrote: post #68
saagarjha wrote: post #65

Earlier quoted context omitted.

In theory, yes. In practice you will probably be forced to tile your GEMM and incur the penalty of redundant memory accesses.

Sure, but still on each tile, you do O(k^3) compute with O(k^2) memory, and you generally arrange things so that at least one tile is in L1 and at least one other is in L2/LLC (using CPU idioms), so again, you have plenty of bandwidth (typical choices of k are in the ballpark of ~32, and a 32:1 compute to memory ratio is just fine on most hardware, especially if some of those accesses are coming from fast memory)

I don't think so? It is too late for me to actually do the math on this but if you take the degenerate case where the tile size is literally 1 element then you will do as many loads as arithmetic operations. Thus I would consider any sort of fixed tiling (which you would be resigned to due to your caches being of limited size) requiring O(n^3) loads?

Re: Optimizing a WebGPU Matmul Kernel for 1 TFLOP

#80

Post by saagarjha »

Const-me wrote: post #69
saagarjha wrote: post #64

Earlier quoted context omitted.

I believe the point being made was that this could be done in the CPU faster than was achieved here.

Yeah, but not on a single core. In my desktop computer, I have Ryzen 7 8700G CPU, which has 8 Zen 4 cores, 4.2 GHz base frequency, 65W TDP. Theoretically, when doing FP32 FMA, each CPU core can do 32 FLOP/cycle. At the base frequency, this translates into 134 GFlops per core. You gonna need all 8 cores to achieve 1 theoretical TFlops. BTW, integrated GPU inside the same 8700G processor can theoretically do 8.2 TFlops…

I am told the numbers above require the core to have a matrix multiply unit (such as SME)
Post reply on HN