The early History of the Singular Value Decomposition (1993) [pdf]
1–10 of 85 posts
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#2The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#3For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.
Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#4For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.
https://www.oceanopticsbook.info/view/photometry-and-visibil...
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#5For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive. https://www.oceanopticsbook.info/view/photometry-and-visibil...
Okay, but that was a really useful metaphor if incomplete in a lot of ways. It made me say “oh”.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#6For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive. https://www.oceanopticsbook.info/view/photometry-and-visibil...
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#7For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…
Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD because it uses a faster method to compute matrix sign. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#8The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
It comes up anywhere that youre working with data that has some sort of correlation structure.
In image processing, the SVD makes it possible to talk about all the rich spatial correlations in the image, and pick out the strongest ones and discard noise.
This is also why it's so ubiquitous in compression algorithms, and of central importance in stuff like quantum information.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#9Earlier quoted context omitted.
Just going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive. https://www.oceanopticsbook.info/view/photometry-and-visibil...
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
Doesn't matter: there's no RGB model that captures the colour space. That exactly the reason CIE exists.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#10The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
what work are you doing in computer vision that isn't entirely ML these days?