Can anyone suggest a starting point to be able to read mathematics papers like this and understand them?
The early History of the Singular Value Decomposition (1993) [pdf]
31–40 of 85 posts
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#32Can anyone suggest a starting point to be able to read mathematics papers like this and understand them?
Axler’s _Linear Algebra Done Right_ should get you at least part of the way there, content-wise, depending on what other math background you have.
As for reading math papers in general, it’s mostly a process of stepping through it incrementally and trying to verify the steps you don’t understand based on the surrounding context. Most of the concepts in this paper are accessible on Wikipedia or elsewhere, you can make small (e.g. 2 x 2) examples as you go and see what happens.
It’s not an easy skill to acquire from scratch, especially from outside the ivory tower.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#33Some fun stuff about SVDs: If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0]. I'm not claiming it's a practical way to do so, but this means that if…
Aren't you skipping over the noise/stochasticity part from the sampling?
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#34Some fun stuff about SVDs: If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0]. I'm not claiming it's a practical way to do so, but this means that if…
Aren't you skipping over the noise/stochasticity part from the sampling?
Yes, that too, in addition to the other differences I pointed out.
But it's just an SVD with a few more bells and whistles in my view.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#35Chapter 7 of Linear Algebra Done Right by Sheldon Axler reads almost as poetry.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#36Some fun stuff about SVDs: If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0]. I'm not claiming it's a practical way to do so, but this means that if…
I also forgot that you have to constrain U and V to be orthonormal - otherwise you end up with the same low rank approximation D', but exactly which numbers go in U, S, or V can shift as it's underdetermined without that constraint.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#37Chapter 7 of Linear Algebra Done Right by Sheldon Axler reads almost as poetry. https://linear.axler.net/
Really? I find the part about the SVD in Axler's book extremely unhelpful, big blobs of opaque formulas and jargon with next to no explanation or context that basically require either knowing the topic fully beforehand or a huge amount of effort to parse.
e.g. Axler's definition of singular values is the extremely dry and technical:
> Suppose T is in L(V, W). The singular values of T are the nonnegative square roots of the eigenvalues of T†T, listed in decreasing order, each included as any times as the dimension of the corresponding eigenspace of T†T.
(Using a dagger instead of an asterisk for the conjugate transpose since HN interprets and asterisk to mean italics.)
If you already just proved a lot of stuff about eigenvalues, this could be a serviceable definition; at any rate it saves space. But it doesn't really explain the point.
I'd recommend anyone interested in this or related topics read Trefethen & Bau (1997) Numerical Linear Algebra.
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#38For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…
Eigenvalue are more flexible than singular values -- If you have procedure for computing eigenvalues of C, squared singular values of X are eigenvalues of C=XX'. Left and write singular vectors of X and eigenvectors of XX' and X'X. For rotation rotation matrix, singular values are all 1. Meanwhile eigenvalues tell you angles of rotation https://math.stackexchange.com/questions/4874616/obtaining-a...
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#39The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
what work are you doing in computer vision that isn't entirely ML these days?
I'm not the person you are replying to but I work in image processing of SAR radar images and it's mostly ML-free (thankfully because I don't enjoy it). I dont know which other areas still work with these things
Re: The early History of the Singular Value Decomposition (1993) [pdf]
#40Earlier quoted context omitted.
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
Doesn't matter: there's no RGB model that captures the colour space. That exactly the reason CIE exists.
You can represent any arbitrary colors using RGB values for whatever "R", "G", and "B" primaries you like, they just might not fit in the range [0, 1].
(At least, under the strong assumptions we make in color modeling using the CIE system of colorimetry; the basic keyword is "Grassmann's laws".)