Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The early History of the Singular Value Decomposition (1993) [pdf]

math.ucdavis.edu

Post reply on HN
Print view View original

71–80 of 85 posts

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#71

Post by FabHK »

muragekibicho wrote: post #3

For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…

My intuitive take:

Eigenvectors answer the question, if you have a linear mapping from a space to itself: which lines through (through the origin) remain unchanged after the mapping? And the associated eigenvalues tell you how much a specific point on that line moves along the line. For example: If you rotate things in 3D, the rotation axis remains unchanged. That's the eigenvector. If you reflect things through a mirror plane, any line in that mirror plane will remain unchanged (with eigenvalue 1). Other lines will be, well, mirrored, but there is one line that is reflected onto itself: the one perpendicular to the mirror plane. However, points on it will go from one side to the other, so have eigenvalue -1.

The SVD says something else: Any linear mapping from a space (to itself or a different one - higher or lower dimensional) can be expressed as 1) first a rotation in the old space, then 2) some coordinate scaling or dilation (expansion/contraction) along the coordinate axes, described by the diagonal matrix with singular values, 3) then another rotation in the new space.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#72

Post by lr1970 »

yaroslavvb wrote: post #53
xscott wrote: post #47

Earlier quoted context omitted.

That's like saying spoons are more flexible than forks because you have soup (rotation matrices). Spoons and forks both work for rice (pos def matrices), and you'll want a fork for noodles (rectangular matrices).

You can use eigenvalue solver to find singular values of any matrix but not vica versa. Implementing eigenvalues needs complex numbers

> Implementing eigenvalues needs complex numbers

This is exactly why eigenvalues are less flexible than singular values. You can have a real valued matrix that does not have any eigenvalues in the field of real numbers, all eigenvalues are complex numbers. Examples: rotation matrixes in R^2 have no eigenvalues in R. Singular values, on the other hand, are always real (they are eigenvalues of the Hermitian matrix MM^*) and can be used the same way for the matrices over real (R) or complex (C) numbers, hence extra flexibility. Added bonus -- singular values are never negative.

Instead of "more flexible" a better statement would be -- eigenvalues convey more information than singular values.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#73

Post by srean »

LPisGood wrote: post #69
tzury wrote: post #61

Earlier quoted context omitted.

They do. You get some in the prose, and then more in the exercises. Learn ____ the hard way.

Or the book could fill in more details with a rich exposition and supplement with high quality exercises.

I have mixed feelings about this. BTW my comment is not about Axler's linear algebra book but math pedagogy in general.

Spoon feeding doesn't really help with learning, internalising, building internalisation for. Effort and frustration is key.

For me the best way has been to be forced to discover the important properties and results myself. It is slow but effective and best for knowledge retention. I think the best way to enforce is exercises with grades of progressive hints.

One of my best teachers used to ask us to prove things that are not true. The effort and frustration to prove those were the best teaching moments. We soon caught on to his method though and we're on our alert whenever we were asked to prove something to be true.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#74

Post by fooblaster »

waynecochran wrote: post #16
fooblaster wrote: post #10

Earlier quoted context omitted.

what work are you doing in computer vision that isn't entirely ML these days?

I'll give you one example, an often first step to solving the Perspective N Point (PNP) problem involves using the Direct Linear Transform (DLT) method which boils down to solving AX = 0 where A in a 12x2N matrix (N can be 6 to 500). The best way to solve this is with SVD. The first published PNP solver (for N = 3) dates to 1841 (did not use SVD) and we still are solving that problem now and I imagine we will still b…

classic! Used the same thing to solve for the rigid transform of an April tag for a calibration problem years ago.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#75

Post by fooblaster »

aptitude_moo wrote: post #39
fooblaster wrote: post #10

Earlier quoted context omitted.

what work are you doing in computer vision that isn't entirely ML these days?

I'm not the person you are replying to but I work in image processing of SAR radar images and it's mostly ML-free (thankfully because I don't enjoy it). I dont know which other areas still work with these things

What sort of algorithms do you run on SAR images?

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#76

Post by waynecochran »

shusaku wrote: post #56
waynecochran wrote: post #24

Earlier quoted context omitted.

Go ahead and have claude add and run units tests for you as part of the PR review process.

Claude is absolutely terrible at writing unit tests, this advice continues to perplex me. Coding agents usually slam you with hundreds of lines that do little more than “test” what is obvious. Maybe it’s good for an API revision or such. But for linear algebra where errors might not manifest unless suitably chosen inputs are generated based on expert knowledge, that’ll be useless.

Have you used Claude lately? It definitely will outperform a human at this task now.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#77

Post by techas »

TimorousBestie wrote: post #11
waynecochran wrote: post #2

The SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.

> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.

Solution of linear system is one of those problems that you should program yourself to learn the methods. But there are so many edge cases and details that you have to rely on libraries for production.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#78

Post by waynecochran »

bee_rider wrote: post #49
waynecochran wrote: post #46

Earlier quoted context omitted.

There are cases where you don't want the footprint of bringing in another framework / library or that is not even an option.

Yeah, I’m curious what put them in that case here, though. There are high quality open source linear algebra codes out there, so I’m surprised that they needed something generated. (But, just surprised and curious to hear more, not saying they are wrong or anything like that).

Here is an example. I wrote a MacOS app in Objective-C++/C++ that used Eigen's JacobiSVD solver at one point. A colleague of mine use Claude to port it into a web tool that used JavaScript. I scrolled thru the JS code and saw that is wrote is own Jacobi SVD solver. So I assume it did this match the exact behavior of the MacOS app -- using a 3rd party solver may not accomplish this. SVD solvers can behave differently in different use cases. What are the options for JS?

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#79

Post by jacobolus »

srean wrote: post #73
LPisGood wrote: post #69

Earlier quoted context omitted.

Or the book could fill in more details with a rich exposition and supplement with high quality exercises.

I have mixed feelings about this. BTW my comment is not about Axler's linear algebra book but math pedagogy in general. Spoon feeding doesn't really help with learning, internalising, building internalisation for. Effort and frustration is key. For me the best way has been to be forced to discover the important properties and results myself. It is slow but effective and best for knowledge retention. I think the best…

The big problem I have with Axler's presentation of the SVD is that it's backwards. It leads with a bunch of completely dry and technical minutiae written formally, loads students up with tedious and confusing technical exercises they aren't likely to appreciate, and defers the motivation, context, explanation, and pictures until a few dozen pages later (probably multiple weeks later for a course), and in my opinion doesn't do a great job with them even then. I think this does a big disservice to students, and I wouldn't recommend anyone learn about the SVD this way. (Admittedly, the SVD is treated more like a curious aside than a centrally important tool in Axler's book; he's not trying to train people to use numerical methods.)

The best place to learn about the SVD is probably in the context of some kind of concrete problem (the most illuminating would be a problem in statistics, image processing, geometry modeling, or whatever, but it could also be a more abstract pure math problem, something about quadratic forms or something) that demonstrates an actual need for it, and then introduce the idea with an intuitive explanation supported by pictures and spatial reasoning. The actual technical details are not really that complicated or hard to figure out once you understand the concept, but if you don't understand the concept then trying to prove a bunch of obscure technical statements just seems like pointless busywork.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#80

Post by bee_rider »

waynecochran wrote: post #78
bee_rider wrote: post #49

Earlier quoted context omitted.

Yeah, I’m curious what put them in that case here, though. There are high quality open source linear algebra codes out there, so I’m surprised that they needed something generated. (But, just surprised and curious to hear more, not saying they are wrong or anything like that).

Here is an example. I wrote a MacOS app in Objective-C++/C++ that used Eigen's JacobiSVD solver at one point. A colleague of mine use Claude to port it into a web tool that used JavaScript. I scrolled thru the JS code and saw that is wrote is own Jacobi SVD solver. So I assume it did this match the exact behavior of the MacOS app -- using a 3rd party solver may not accomplish this. SVD solvers can behave differently…

Not sure, I’m not familiar with the web ecosystem. Not familiar at all, so my next question will probably be stupid. Would it have been possible to compile your C++ library to WebAssembly?
Post reply on HN