One geometric idea that explains SGD, AdamW, and Muon
kbwal.github.io
One geometric idea that explains SGD, AdamW, and Muon
1–5 of 5 posts
Re: One geometric idea that explains SGD, AdamW, and Muon
#2Wow awesome!!
Re: One geometric idea that explains SGD, AdamW, and Muon
#3Quite fascinating!
Re: One geometric idea that explains SGD, AdamW, and Muon
#4What are your thoughts on some of the disadvantages of using Muon over AdamW?
Re: One geometric idea that explains SGD, AdamW, and Muon
#5What are your thoughts on some of the disadvantages of using Muon over AdamW?
I'm honestly not too aware of the true scene of which optimizers are used nowadays by which labs (though I think Moonshot has shown Muon's ability to scale to large models, and Muon is definitely more used now).
That being said, obviously orthogonalization is pretty expensive! I looked it up and it's roughly 5-15% slower per step, but it converges in fewer steps fwiw. I'm guessing it's also less battle-tested than AdamW is, so AdamW still might be the safer option (and you're gonna have to use AdamW for your 1D params anyways).