Trends in Machine Learning Hardware

TLDR: the H100, lower precision, and other advances lead to a big jump in computational performance. We're in for a wild ride when the next generation of models is trained on 100x more compute in 2024 and 2025.