A Programmer’s Intuition for Matrix Multiplication • Hacker News Board

Re: A Programmer’s Intuition for Matrix Multiplication

#91

Post by markisus » Fri, Oct 23, 2020, 12:23 PM UTC

OneGuy123 wrote: post #43

If we're talking about 3D rendering then a matrix multiplication with a vector are just projections: A 3x3 matrix that transforms a vector is nothing else than that vector being projected onto the 3 axis which are inside the 3x3 matrix. This is very easy to see visually. A matrix times matrix multiplication (eg 3x3 times 3x3) is just projecting the axis of one matrix onto the other: expressing the coordinate system i…

The projection viewpoint for a rotation matrix using dot products is totally valid, but I think the column viewpoint is more intuitive.

A 3x3 matrix can be viewed as something that acts on the standard unit basis vectors. The action on the standard unit basis vectors can be read out by looking at the columns. (A matrix times the ith standard basis vector is just the ith column.) Since the transform is linear, all the other points in space are carried along with the action as well. This is the "linear map" viewpoint of a rotation matrix.

I show an example here https://biro.ai/we-learned-the-wrong-way-to-matrix-multiply/ where this viewpoint lets you instantly see what a rotation matrix does by looking at the columns.

Re: A Programmer’s Intuition for Matrix Multiplication

#92

Post by Tainnor » Fri, Oct 23, 2020, 1:16 PM UTC

rathel wrote: post #87

Vespasian wrote: post #86

Earlier quoted context omitted.

I can wholeheartedly recommend his channel. In particular his recent "Lockdown Math" series taught me a lot. I knew and used trigonometric functions and exponentials before but they never really "clicked". Grant is a tremendous educator.

True, but I don't agree with what he said here @ 8:00 https://youtu.be/ZxYOEwM6Wbk?t=481 Is this really the case? I checked Wikipedia and MathWorld and nobody makes a distinction between e^x and exp(x). Even if it's a "white lie" for didactic reasons, I don't buy it, it will be much more confusing down the line for students. Math is about finding structures governed by some rules and then generalizing them. To me the…

I think the way he said it is a bit weird (and definitely nobody would think of e^x and exp(x) ss being two different things). But I would agree that it's probably best to define e^x through its power series, especially because it directly generalises to complex arguments. (Another good definition is as the function f whose derivative equals itself, and which satisfies f(0)=1, but that's not very constructive.)

Of course, you can also go another route: you can define the number e (e.g. as the limit of (1+1/n)^n), then you can straightforwardly define natural number powers of e by repeated multiplication, then integer and rational powers through reciprocals and roots (you have to prove n-th roots exist, but that's doable), and then you can define real number exponents via limits (maybe this is what you mean by "interpolating recursively"). Now, when we come to complex numbers, you can use Euler's formula as a definition instead of a theorem, and define exp(a+bi):=e^a * (cos b + i sin b). Of course, at the end of this whole exercise you can prove that this definition is exactly equal to the power series definition.

Another benefit of the power series definition is that it also generalises e.g. to the matrix exponential (exp(A), when A is a matrix).

Re: A Programmer’s Intuition for Matrix Multiplication

#93

Post by Tainnor » Fri, Oct 23, 2020, 1:27 PM UTC

crdrost wrote: post #84

pmiller2 wrote: post #78

Earlier quoted context omitted.

If A is a square matrix, then a left eigenvector v is a vector such that vA = \lambda_v v for some \lambda_v. Likewise, if u is a right eigenvector of A, Au = \lambda_u u. Notice that u and v cannot be equal, because they are not the same shape. However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. More or less what this means is that we tend to…

> However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. Consider the matrix M = [ 0.50 0.50 ] [ 0.25 0.75 ] Clearly [1; 1] is a right eigenvector with eigenvalue 1. But [1 1] M = [0.75 1.25]. > A matrix does not necessarily have nontrivial eigenvectors. Think about the 0 matrix here. The zero matrix has _all_ the nontrivial vectors as eigenvecto…

> I don't know why you consider the eigenvalue zero to somehow not count as an eigenvalue. Very suspicious.

Zero can of course be a proper eigenvalue. Probably they confused it with the fact that the zero vector typically doesn't count as an eigenvector.

Re: A Programmer’s Intuition for Matrix Multiplication

#94

Post by Tainnor » Fri, Oct 23, 2020, 1:33 PM UTC

crdrost wrote: post #85

philip-b wrote: post #81

Earlier quoted context omitted.

>But, if A is nonzero, then it must have at least one nonzero eigenvalue, hence one nontrivial eigenvector. How about this matrix? [[0, -1], [[1, 0]] >However, if v is a right eigenvector with eigenvalue \lambda, then v^T is a left eigenvector with eigenvalue \lambda, as well. A snippet of code producing a counterexample: import numpy as np import scipy.linalg as spla A = np.random.randn(3, 3) right_eigenvector = spl…

Your first matrix doesn't count, but the theorem is bogus: see my counterexample above. The correct theorem is that every complex square matrix has an eigenvector. If we interpret your matrix as a complex matrix, then it does have two eigenvectors, namely [1; ±i]. Hence why I’d say it kind of “doesn’t count.”

The way to see it is to realise that the eigenvalues are the roots of the characteristic polynomial (which is not hard to prove). But by the fundamental theorem of algebra, every complex polynomial has a root (it does actually have n roots, where nxn is the dimension of the matrix, but those roots can all be the same). This also shows why the theorem is, in general, false if you restrict yourself to real numbers.

(I also think that the counterexample exhibited "counts" in the way that it's intuitively very clear why it can't have a real eigenvector. The R^2 plane can be easily visualised, as opposed to C^2, and the matrix exhibited acts as a rotation of R^2, so of course there can't be an eigenvector. As you said, once you allow complex numbers, that's not true anymore, but at that point the visual intuition breaks down somewhat.)

Re: A Programmer’s Intuition for Matrix Multiplication

#95

Post by tsjq » Sun, Oct 25, 2020, 6:29 PM UTC

hchasestevens wrote: post #20

I've always found the bipartite graph conceptualization of matrix multiplication the most intuitive, and especially so if you're familiar with neural networks: https://www.math3ma.com/blog/matrices-probability-graphs

Thanks for linking to this.