Quaternions are beautiful too until you sit down to multiply them.
I’ve been swimming in quaternions all week. Thank you for that :D
Is Matrix Multiplication Ugly?
111–113 of 113 posts
Re: Is Matrix Multiplication Ugly?
#112Earlier quoted context omitted.
A somewhat more beautiful matmul for neural networks is given by the Monarch paper: https://arxiv.org/abs/2204.00595 Generally, low-rank and block-diagonal matrices are both great strategies for producing expressive matmuls with fewer parameters. We can view the FFT as a particularly deft example of factorizing one big matmul into a number of block-diagonal matmuls, greatly reducing the overall number of multiplicati…
Do low rank/block diagonal matrices come up in LLMs often? What about banded or block tridiagonal? Intuitively banded matrices seem like they ought to be good at encoding things about the world… everything is connected but not randomly so.
Yep!
Think of LORA for network fine tuning. Monarch (linked above) uses lots of block diagonality. These ideas also make flash attention flash.
I haven't seen banded matrices as much, though (with weight sharing) they're just convolutions. One nice feature of block diagonality is that you can express it as batched matrix multiplication, reusing all the existing matmul kernels.
Re: Is Matrix Multiplication Ugly?
#113Earlier quoted context omitted.
> sends the pair (x, y) to the pair (−x, y) I know linear algebra, but this part seems profoundly unclear. What does "send" mean? Following with different examples in 2 by 2 notation only makes it worse. It seems like you're changing referents constantly.
I’ve updated this passage. Let me know if the new version is clearer.
> that, for all x and y between −1 and 1, changes the sign of x
Perfect! Overall, it's much better (though I really just meant that "send" was vague, which "change" improves)!