Could something like this be done in WebGPU?
CubeCL supports WebGPU and can be used with wasm!
Multiplatform Matrix Multiplication Kernels
21–30 of 32 posts
Re: Multiplatform Matrix Multiplication Kernels
#22I had bet that matmult would be in transformer-optimized hardware costing a fraction of GPUs first class in torch 2 years ago with no reason to use GPUs any more. Wrong.
> matmult would be in transformer-optimized hardware It is... it's in GPUs lol > first class in torch It is > costing a fraction of GPUs Why would anyone give you this for cheaper than GPUs lol?
I think they’re referring to hardware like TPUs and other ASICs. Which also exist, of course :)
Re: Multiplatform Matrix Multiplication Kernels
#23Earlier quoted context omitted.
> matmult would be in transformer-optimized hardware It is... it's in GPUs lol > first class in torch It is > costing a fraction of GPUs Why would anyone give you this for cheaper than GPUs lol?
I think they’re referring to hardware like TPUs and other ASICs. Which also exist, of course :)
Sure but GPUs literally have MMA engines now
Re: Multiplatform Matrix Multiplication Kernels
#24GPUs came about because of the need for faster float 4x4 and 3x3 matrix, and 3 and 4 vector math ops like multiply, multiply-accumulate, and such, and faster pushing of pixels with things like texture mapping. All hail OpenGL and dual Voodoo2 SLI. ;)
Re: Multiplatform Matrix Multiplication Kernels
#25One of the author here, don't hesitate if you have any question or comment!
Reminds me of ye olden days when kernel transforms were merely weighted multiplicative and/or additive matrixes applied to every point in the source arriving at pixel data in the target. Blur, sharpen, color channel filter, color swap, invert, etc. An extremely diagonalizable problem suitable for massive parallelism and concurrent calculation because there is little/no dependency on prior calculations.
Re: Multiplatform Matrix Multiplication Kernels
#26I had bet that matmult would be in transformer-optimized hardware costing a fraction of GPUs first class in torch 2 years ago with no reason to use GPUs any more. Wrong.
The real bottleneck is the memory, optimize your matmul architecture all you like whilst you still have it connected to a big chunk of HBM memory (or whatever your chosen high bandwidth memory is) you can only do so much.
So really GPU v not GPU (e.g. TPU) doesn't matter a whole lot if you've got fundamentally the same memory architecture.
Re: Multiplatform Matrix Multiplication Kernels
#27I'm sorry this is a low brow comment but this is the dumbest thing you can do in this space: > Unit (thread in CUDA, invocation in Vulkan/Wgpu): the smallest execution entity performing computations. > Plane (warp in CUDA, subgroup in Vulkan/Wgpu): a group of (typically 32) units executing in lockstep and able to share data efficiently through registers. > Cube (thread block in CUDA, workgroup in Vulkan/Wgpu): a grou…
Relevant XKCD: https://xkcd.com/927/
Re: Multiplatform Matrix Multiplication Kernels
#28Has there been much research into slightly flawed matrix multiplications? If you have a measure of correctness, and a measure of performance. Is there a maximum value of correctness per some unit of processing that exists below a full matrix multiply Obviously it can be done with precision, since that is what floating point is. But is there anything where you can save x% of computation and have fewer than x% incorrec…
If you do it in 8-bit it's usually 2x as fast as 16 bit on Tensorcores
Re: Multiplatform Matrix Multiplication Kernels
#29Has there been much research into slightly flawed matrix multiplications? If you have a measure of correctness, and a measure of performance. Is there a maximum value of correctness per some unit of processing that exists below a full matrix multiply Obviously it can be done with precision, since that is what floating point is. But is there anything where you can save x% of computation and have fewer than x% incorrec…
Well, approximate computing seems to be a superset of the field you describe here, with many different approaches, including analog computation. As you say, some algorithms care a bit less about precision, especially for LSBs.
Re: Multiplatform Matrix Multiplication Kernels
#30Seems kind of CUTLASS-inspired in terms of how its API is designed. I'm curious how they plan to expose more interesting operations, though, since CUTLASS gives you the ability to write custom epilogs and tiling patterns, and I'm not sure their API is expressive enough to do this.