61–70 of 84 posts
Boo LU!!! Go with stable QR!
But otherwise I agree, QR is almost better on every front.
Thank you for this wonderfully concise summary: it’s convenient to have all this in one compact document. I suppose “flops” means “floating-point operations” here? Heretofore I’ve always encountered this as an abbreviation for “floating-point operations per second”.
Any suggestions on what to learn in Linear Algebra after Gilbert Strang’s 18.06SC? https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011... My goal is to learn the math behind machine learning.
For most current gen deep learning there's not even that much math, it's mostly a growing library of what are basically engineering tricks. For example an LSTM is almost exclusively very basic linear algebra with some calculus used to optimize it. But by it's nature the calculus can't be done by hand and the actual implementation of all that basic linear algebra is tricky and takes practice.
You'll learn more by implementing things from scratch based on the math than you will trying to read through all the background material hoping that one day it will all make sense. It only ever makes sense when implementing and by continuous reading/practice.
Spectral decomposition is pretty cool. Application 1 - spectral clustering - an alternative to k-means for nonlinear clusters. Get a Distance matrix of your data, spectral decomp, run k-means on your k top eigen vectors and that's your clusters. Application 2 - graph clustering - (run spectral clustering on adj matrix!) There's some tricks to getting it to work in practice like normalizing but it's a simple and power…
I've never given a second thought about what the etymology of "spectral" in spectral decomposition is. Somewhere in the back of my mind (and I guess many students of physics have the same notion) subconsciously i assumed it originates from eigenvalues of the Hamiltonian determining the atomic spectral lines . But I've never followed up on it and actually looked it up .
But the way I see it "spectral decomposition of A" is a way to express A as a sum of orthogonal, rank-1, operators. A = \sum l_i u_i u_i^T. Those l_i are the eigenvalues; u_i are the eigenvectors.
The eigenvectors look a whole lot like the "modes" in a Fourier decomposition. And if you plot (i, l_i), the eigenvalues are a bit like the "spectrum" (the amplitude of each mode).
In fact, the complex exponentials (the modes in the Fourier decomposition) are also eigenvectors of a specific operator (the Laplacian).
Math people are good at finding connections between things.
Spectral decomposition is pretty cool. Application 1 - spectral clustering - an alternative to k-means for nonlinear clusters. Get a Distance matrix of your data, spectral decomp, run k-means on your k top eigen vectors and that's your clusters. Application 2 - graph clustering - (run spectral clustering on adj matrix!) There's some tricks to getting it to work in practice like normalizing but it's a simple and power…
I've never given a second thought about what the etymology of "spectral" in spectral decomposition is. Somewhere in the back of my mind (and I guess many students of physics have the same notion) subconsciously i assumed it originates from eigenvalues of the Hamiltonian determining the atomic spectral lines . But I've never followed up on it and actually looked it up .
Earlier quoted context omitted.
I've never given a second thought about what the etymology of "spectral" in spectral decomposition is. Somewhere in the back of my mind (and I guess many students of physics have the same notion) subconsciously i assumed it originates from eigenvalues of the Hamiltonian determining the atomic spectral lines . But I've never followed up on it and actually looked it up .
I might be wrong about the exact historical reason. But the way I see it "spectral decomposition of A" is a way to express A as a sum of orthogonal, rank-1, operators. A = \sum l_i u_i u_i^T. Those l_i are the eigenvalues; u_i are the eigenvectors. The eigenvectors look a whole lot like the "modes" in a Fourier decomposition. And if you plot (i, l_i), the eigenvalues are a bit like the "spectrum" (the amplitude of ea…
From a theoretical perspective the most fundamental decomposition is the rank-decomposition: A = X D Y with X,Y invertible and D diagonal. It's called rank decomposition, because you can read off the rank from A by counting the non-zero entries in D. It's also useful to determining bases for the image and the kernel of A. Every math student learns a version of that in their first lecture series of Linear Algebra. Cur…
As it turns out, the author does cover the rank-decomposition (under the name "Rank-Revealing Factorization") in his blog as well: https://nhigham.com/2021/05/19/what-is-a-rank-revealing-fact...
Now apply them to the data used to identify the "Big Five" personality traits! This is an interesting application of factor analysis/matrix factorization: https://en.wikipedia.org/wiki/Big_Five_personality_traits
The SVD is everywhere. IIRC, the first Netflix price was one won with the SVD:
https://en.wikipedia.org/wiki/Netflix_Prize
https://cseweb.ucsd.edu/classes/fa17/cse291-b/reading/Progre...
This is a fantastically clear outline of this topic. Thank you! > The terms “factorization” and “decomposition” are synonymous and it is a matter of convention which is used. Our list comprises three factorization and three decompositions. I can't tell if this is a joke: right after saying that these two words mean the same thing in this context, they are then used to categorize the methods. Edit: This is the kind of…