Matrix Orthogonalization Improves Memory in Recurrent Models • Hacker News Board

Re: Matrix Orthogonalization Improves Memory in Recurrent Models

#31

Post by impossiblefork » Wed, Jul 01, 2026, 8:38 PM UTC

phkahler wrote: post #10

If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.

I wouldn't say that making the matrix diagonal in some basis is some further step.

If we have an singular value decomposition, M=USV^*, the columns of U are linearly independent they are a basis for the space M maps things into, and the columns of V are linearly independent then it's a basis for the space it maps things from, and [M]_{BB'} = S.

Re: Matrix Orthogonalization Improves Memory in Recurrent Models

#32

Post by wwweston » Thu, Jul 02, 2026, 1:19 AM UTC

cyanydeez wrote: post #30

seanhunter wrote: post #29

Earlier quoted context omitted.

Definitely. It’s a lesson for me in remaining humble and not making too many confident predictions.

while cute, that doesn't address the size and magnitude of the "AGI" and Singularity that AI proponents claim, and definitely not the person with anxiety that they're some how going to be put into the "permanent underclass" Another good line to look at is how people believe in ghosts: people with established religions without "ghosts" are less likely to believe in ghosts than people with atheism, even when they'd sup…

I’ve totally lost the thread here but this is interesting.

Who are the religions without ghosts?

And is the overall point “just because we’ve made big leaps of progress doesn’t mean every challenge is tractable let alone a moonshot sprint away especially those we have solid theoretical limits on”? That’s a point I certainly find amenable to I just want to make sure I’m not missing something more subtle or sophisticated.

Re: Matrix Orthogonalization Improves Memory in Recurrent Models

#33

Post by phkahler » Thu, Jul 02, 2026, 1:23 PM UTC

bee_rider wrote: post #17

phkahler wrote: post #10

If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.

I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can always do that, but I’m sure that’s been tried already.

But orthogonal matrices are square.

Re: Matrix Orthogonalization Improves Memory in Recurrent Models

#34

Post by bee_rider » Thu, Jul 02, 2026, 5:50 PM UTC

phkahler wrote: post #33

bee_rider wrote: post #17

Earlier quoted context omitted.

I don’t know AI, but, weight matrices aren’t square in general, right? My first guess for something like this would be to take the SVD instead, since you can always do that, but I’m sure that’s been tried already.

But orthogonal matrices are square.

Ah, good point. I assumed they were coming up with some orthogonal basis vectors for a (potentially) non-square matrix. Edit: actually I’m not sure, this Newton-Schulz process seems to work for non-square matrices as well. Generally I see “orthogonalization” refer to the process of coming up with those orthogonal basis vectors but it could be a domain-specific lingo thing.