Viewing profile — kz919

"AdamW has been the default optimizer for transformer pretraining. For many years, our community searches for faster and more stable optimizers with only constraint positive outcom…

obviously because they can't beat it... There will be zero reason to use it when you have better transformer based models that can fit the existing infrastructure.

you are not wrong, but I don't think that is their point. It's just a muscle flex to shows you that their hardwares work. If it can train 176B or whatever, it should be an easy pea…

you just need to teach it to use a calculator https://arxiv.org/pdf/2302.04761.pdf

I wonder how consistent the chip output would be. Also compiler and software stack sound painful to build.