21–30 of 34 posts
I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.
I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix. I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.
Here is a pytorch optimizer that can maintain a matrix as orthogonal throughout optimization: https://github.com/adrianjav/pogo — POGO: A Proximal One-step Geometric Orthoptimizer https://arxiv.org/abs/2602.14656 — An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale; Adrián Javaloy, Antonio Vergari
I can't help but think of orthogonal frequency-division multiplexing and it's use in encoding data on multiple carrier frequencies, and it makes me wonder what other parallels we will discover between digital transmission technology for cross-domain stuff like this.
I have this strange sensation that I can't put into words that somehow we are on the brink of unveiling an entirely new paradigm of AIs or perhaps even of combining AI with classical algorithms in a way to rapidly iterate between each other (and sensor data) that will instantly 10x or 100x current capabilities. Anyone else feel this?
In the 1920s we had legions of very smart, highly trained (arguably better trained in mathematics) basically chucking relays and vacuum tubes together with reckless abandon to build the most valuable and complicated systems mankind had ever come up with (telephony, radio, radar, etc). They had no idea how they worked and only ad-hoc rules of thumb to construct them.
It took the insight of a handful of these people both in and outside of industry to formalize the theory of operation of most of what people were already building and then use that theory to establish formal design practices.
The people before these theories were realized were exceptionally smart and good at what they did, it's just they didn't have better design tools to reason about the things they were building.
And once they had those tools they didn't 10x or 100x overnight.
If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.
Now I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have different personalities?
Neural networks are non-linear, so I think you wouldn’t be able to compute typical eigenvalues. You could compute the eigenvalues and/or singular of the individual weight matrices (I’m sure this has been studied). SVDs are very conventional for making low-rank approximations, so it must have been studied. The concept of nonlinear eigenvalues exists, but it is a bit more exotic.
Someone found a way to get "something like" a tri-diagonal matrix that was equivalent to the LLM they were studying in 2022.
Apologies for being informal and hand-wavey. Been a long time and I probably forgot a few important points.
I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix. I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.
The Newton-Schulz iteration they use approximates setting all singular values of the matrix to 1. That computes the nearest orthogonal matrix under the Frobenius norm.
I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix. I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.
I wonder if there's a similar shortcut representation that we will eventually realize we should be using for ML. I suppose if there is one, it won't have native GPU support, so no one will bother looking for it.
Earlier quoted context omitted.
That wasn’t just a physics professor that was William Thompson aka Lord Kelvin (the dude the temperature unit is named after and one of the most important mathematical physicists of the 19th century [1]), who also said that heavier than air flight was physically impossible only a couple of weeks before the Wright Brothers (and presumably in spite of having at least once in his lifetime seen a bird). Proof that you ca…
I love these arguments "You know, we thought we couldn't cross the ocean, and now we did!" This means we can just jump over to mars, then explore other planets, etc, etc. We know tons of regimes where there is non-continuous progress. Finding a smart dude with an anecdote does not invalidate the breadth and width of all human experience with non-continuous systems. Some dude thought all fluid was newtonian, and then…
Earlier quoted context omitted.
I love these arguments "You know, we thought we couldn't cross the ocean, and now we did!" This means we can just jump over to mars, then explore other planets, etc, etc. We know tons of regimes where there is non-continuous progress. Finding a smart dude with an anecdote does not invalidate the breadth and width of all human experience with non-continuous systems. Some dude thought all fluid was newtonian, and then…
Definitely. It’s a lesson for me in remaining humble and not making too many confident predictions.
Another good line to look at is how people believe in ghosts: people with established religions without "ghosts" are less likely to believe in ghosts than people with atheism, even when they'd supposedly be skeptics superstitious claims.
Having functional paradigms is important, and being confident that there isn't a magical extrapolation into AGI is healthier than there being some magical exponential increase that you have to ride the dragon.
Sorry man, we're not solipsistic here. There are reasonable beliefs that are justifiable, instructive and then there are ones that require cherry picking technology indistinguishable from magic without reference to reality and physics.