Earlier quoted context omitted.
The raw model scale is not increasing by much lately. AI companies are constrained by what fits in this generation of hardware, and waiting for the next generation to become available. Models that are much larger than the current frontier are still too expensive to train, and far too expensive to serve them en masse. In the meanwhile, "better data", "better training methods" and "more training compute" are the main w…
> AI companies are constrained by what fits in this generation of hardware, and waiting for the next generation to become available. Does this apply to Google that is using custom built TPUs while everyone else uses stock Nvidia?
If Google wants anything better than that? They, too, have to wait for the new hardware to arrive. Chips have a lead time - they may be your own designs, but you can't just wish them into existence.