Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The early History of the Singular Value Decomposition (1993) [pdf]

math.ucdavis.edu

Post reply on HN
Print view View original

41–50 of 85 posts

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#41

Post by amarcheschi »

jmalicki wrote: post #15

Some fun stuff about SVDs: If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0]. I'm not claiming it's a practical way to do so, but this means that if…

I've recently passed my deep and generative learning exam and in fact before you said vae something was resonating with me, although in our course it wasn't made explicit we were approximating an svd.

Staying in the field of autoencoder, it blows my mind how you can pass from denoising autoencoder (computer science) to scores and eventually matching flows (physics) quite seamlessly

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#42

Post by waynecochran »

jmalicki wrote: post #26
waynecochran wrote: post #24

Earlier quoted context omitted.

Go ahead and have claude add and run units tests for you as part of the PR review process.

That's on the submitter of the PR to do. Don't put that burden on the reviewer.

No problem for claude. In fact I would have claude do the PR and go have lunch.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#43

Post by FabHK »

BTW, if you wonder about the dedication ("For Gene Golub on his 15th birthday"):

Gene Golub was a numerical analyst, and father of the practical singular value decomposition (his license plate read "Prof SVD"), together with William Kahan (the father of IEEE 754 floating point numbers).

And his birthday was February 29. (In other words, the article was on the occasion of Gene's 60th birthday).

https://en.wikipedia.org/wiki/Gene_H._Golub

RIP.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#44

Post by Diogenesian »

snalty wrote: post #31

Can anyone suggest a starting point to be able to read mathematics papers like this and understand them?

This paper is actually not a bad one to get started with. Obviously you need the proper background and I am not sure how much linear algebra you've taken. Going through a senior undergrad numerical linear algebra text might be a good way to learn the prerequsites. In particular, you'll never be able to read a math paper without developing mathematical maturity via textbook exercises.

Assuming you have the background, the boring answer is "patience and practice." For years I had to practically rewrite math papers word for word in order to get anything to stick. These days I am better at reading "mentally," but still sloppy and prone to misreading (just yesterday I misread GPT's proof because I was lazy and on my phone). More so than the empirical sciences, mathematics demands you understand every sentence before moving on to the next. Skimming does you no good. It really does just take patience and perseverance.

The nice thing about this paper is that the math isn't especially advanced, and it's broken up with qualitative historical discussions. If you know a decent bit of linear algebra (enough to understand artificial neural networks), I think you can muddle through this.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#45

Post by jmalicki »

jmalicki wrote: post #15

Some fun stuff about SVDs: If you want to take a low rank approximation to a matrix D, let's call our approximation D'. The approximation that minimizes mean square error of the reconstructed matrix vs. the original (i.e. ||D - D'||_F, the Frobenius norm of their differences) happens to be the truncated SVD, by the Eckart–Young–Mirsky theorem [0]. I'm not claiming it's a practical way to do so, but this means that if…

I made a demo of the SVD part.

https://jmalicki.github.io/svd-grad/

The devil was in the details, and that escalated quickly from a simple idea to actually getting it to work sucked me into a ton of random deep corners.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#46

Post by waynecochran »

bee_rider wrote: post #18
TimorousBestie wrote: post #11

Earlier quoted context omitted.

> Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. I find this so annoying. I had to PR some Claude-generated gaussian elimination routine last month and making sure it got the pivoting logic correct was a waste of my time.

How big of an advantage was it, to have the code developed specifically for your project? These AI tools are pretty impressive, but why not have it generate a call to BLAS or PARDISO or something?

There are cases where you don't want the footprint of bringing in another framework / library or that is not even an option.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#47

Post by xscott »

yaroslavvb wrote: post #38
muragekibicho wrote: post #3

For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues. Singular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys. Optimizers like Muon and Adam play around with weights' first, or second order singular values to train m…

Eigenvalue are more flexible than singular values -- If you have procedure for computing eigenvalues of C, squared singular values of X are eigenvalues of C=XX'. Left and write singular vectors of X and eigenvectors of XX' and X'X. For rotation rotation matrix, singular values are all 1. Meanwhile eigenvalues tell you angles of rotation https://math.stackexchange.com/questions/4874616/obtaining-a...

That's like saying spoons are more flexible than forks because you have soup (rotation matrices). Spoons and forks both work for rice (pos def matrices), and you'll want a fork for noodles (rectangular matrices).

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#48

Post by tzury »

jacobolus wrote: post #37
tzury wrote: post #35

Chapter 7 of Linear Algebra Done Right by Sheldon Axler reads almost as poetry. https://linear.axler.net/

Really? I find the part about the SVD in Axler's book extremely unhelpful, big blobs of opaque formulas and jargon with next to no explanation or context that basically require either knowing the topic fully beforehand or a huge amount of effort to parse. e.g. Axler's definition of singular values is the extremely dry and technical: > Suppose T is in L(V, W). The singular values of T are the nonnegative square roots…

From the preface

“You cannot read mathematics the way you read a novel.

If you zip through a page in less than an hour, you are probably going too fast.

When you encounter the phrase “as you should verify”, you should indeed do the verification, which will usually require some writing on your part.

When steps are left out, you need to supply the missing pieces.

You should ponder and internalize each definition.

For each theorem, you should seek examples to show why each hypothesis is necessary.”

It is a math studying book, not a let me chew it for you before you take it in type of a book. It requires effort, focus and missing steps are missing on purpose so you will discover them. This is the fun about studying mathematics.

Sorry you fell that way, but the book in my opinion is a masterpiece in math composition.

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#49

Post by bee_rider »

waynecochran wrote: post #46
bee_rider wrote: post #18

Earlier quoted context omitted.

How big of an advantage was it, to have the code developed specifically for your project? These AI tools are pretty impressive, but why not have it generate a call to BLAS or PARDISO or something?

There are cases where you don't want the footprint of bringing in another framework / library or that is not even an option.

Yeah, I’m curious what put them in that case here, though. There are high quality open source linear algebra codes out there, so I’m surprised that they needed something generated. (But, just surprised and curious to hear more, not saying they are wrong or anything like that).

Re: The early History of the Singular Value Decomposition (1993) [pdf]

#50

Post by danhorner »

waynecochran wrote: post #42
jmalicki wrote: post #26

Earlier quoted context omitted.

That's on the submitter of the PR to do. Don't put that burden on the reviewer.

No problem for claude. In fact I would have claude do the PR and go have lunch.

Claude can’t have lunch yet.
Post reply on HN