Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Programmer’s Intuition for Matrix Multiplication

betterexplained.com

Post reply on HN
Print view View original

61–70 of 95 posts

Re: A Programmer’s Intuition for Matrix Multiplication

#61

Post by tonyarkles »

layoutIfNeeded wrote: post #52
crdrost wrote: post #22

Earlier quoted context omitted.

That is genuinely a really fun way to look at it, thank you for linking this! Especially this fits nicely with Markov matrices where you have N input nodes and N output nodes and the sum of all of the probabilities coming out of one of the nodes needs to equal 1. What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors, but I suppose that's to be expected—nothin…

>What I might find a little more difficult to teach to people through this lens is the phenomenon of eigenvectors Why? Eigenvectors are simply inputs to the network where the output keeps its shape, that is, at most it gets rescaled, as if you had applied a uniform gain to the components, but otherwise it will be the same as the input.

Oh. Oh my. I have studied linear algebra theory some. I have used it a ton. I have used eigenvectors to solve problems. I have never grokked them.

This description changed that. Thank you!

Re: A Programmer’s Intuition for Matrix Multiplication

#62

Post by da39a3ee »

> However, sometimes the matrix being operated on is not a linear operation, but a set of vectors or data points.

In Jeremy Kun's book [1] he argues that in fact data matrices _can_ be viewed as linear transformations. See e.g. section 10.9 on the SVD

> That is, we’re saying the input is R3 and the basis vectors are people:...

> By representing the ratings this way, we’re imposing the hypothesis that the _process_ of rating movies is linear in nature. ...

[1] https://pimbook.org/

Re: A Programmer’s Intuition for Matrix Multiplication

#63

Post by jpeloquin »

dnautics wrote: post #25
WJW wrote: post #3

Earlier quoted context omitted.

Purist test: aren't vectors just Nx1 matrices? Joking aside, matrices are linear maps in the context of multiplication. They can absolutely have meanings assigned outside the usual "when multiplied by a vector" type of calculations. For example the "adjacency matrix" representation of a graph is very often never multiplied by a vector even in academic algorithm descriptions. YMMV if you call it a (lookup) table in th…

> Purist test The short answer is. Yes, but if your PL treats it that way you will wind up in trouble [0]. The long answer is: https://www.youtube.com/watch?v=C2RO34b_oPM (but seriously take the time to watch this, it's fantastic). [0] Separately C- and often C-derived languages/frameworks like numpy also treat them as 1xN matrices, but that's a nearly tabs-vs-spaces level religious war, with actual consequences. I t…

> numpy also treat them [vectors] as 1xN matrices

I may be misunderstanding something, but in numpy at least a vector, a 1×N array ("row vector"), and a N×1 array ("column vector") behave differently.

  vec = numpy.array([1,2,3])
  assert vec.shape == (3,)
  assert vec.ndim == 1
  row = np.array([[1,2,3]])
  assert row.shape == (1,3)
  assert row.ndim == 2
  col = np.array([[1], [2], [3]])
  assert col.shape == (3,1)
  assert col.ndim == 2
And a vector, unlike a 1×N or N×1 array, can be multiplied to either side of any matrix with compatible cardinality:

  assert all(vec @ numpy.eye(3) == numpy.array([1, 2, 3]))
  assert all(numpy.eye(3) @ vec == numpy.array([1, 2, 3]))
Matlab / Octave, in contrast, forces the user to pick a 1×N or N×1 matrix to represent a vector. Its arrays are always ≥ 2D. So you're probably still right that people get confused translating this to Python. Just wanted to clarify the numpy side of things.

The JuliaCon talk you linked does a great job of explaining what different fields mean by "vector"; thanks for linking it.

Re: A Programmer’s Intuition for Matrix Multiplication

#64

Post by rademacher »

I was expecting this to be more about writing fast matrix multiplies. Surprisingly, it ends up being a lot more than just a triple loop and there's a reason that everyone uses existing libraries like OpenBLAS and MKL [1].

[1] https://www.cs.utexas.edu/users/flame/laff/pfhp/

Re: A Programmer’s Intuition for Matrix Multiplication

#70

Post by jacobolus »

activatedgeek wrote: post #36
adamnemecek wrote: post #12

Matrix multiplication is like convolution.

To be sure there isn't a perspective here that I may be unaware of, is this a reference to circulant matrices, which bear the Toeplitz sparsity structure?

You might find something interesting in https://cr.yp.to/lineartime/multapps-20080515.pdf
Post reply on HN