31–32 of 32 posts
Typically, you are not doing a matrix multiplication for the sake of it, but as part of a broader algorithm (e.g. a simulation). Without fusing those other operations into the MatMul kernel, you are leaving performance on the table. How will the Burn devs address this?
Has there been much research into slightly flawed matrix multiplications? If you have a measure of correctness, and a measure of performance. Is there a maximum value of correctness per some unit of processing that exists below a full matrix multiply Obviously it can be done with precision, since that is what floating point is. But is there anything where you can save x% of computation and have fewer than x% incorrec…
https://kolinko.github.io/effort/
For semi-random weights you cam get down to 20-30% multiplications/mem reads and maintain ~0.98 cosine similarity output between the approximated and full result.
As far as LLM inference goes, the speedup from removing multiplications is at best comparable to the speedup of quantisation (that is - you get at best similar KL divergence score whether you remove calculations or quantise).