Earlier quoted context omitted.
I think you fundamentally don't understand the nature of exponential growth, and the power of diminishing returns. Even if you double the GPU capacity over the next year, you won't even remotely begin to come close enough to producing a step-level growth of capability such as what we experienced between 2 to 3, or even 3 to 4. The LLM concept can only take you so far, and we're approaching the limits of what an LLM i…
>The LLM concept can only take you so far, and we're approaching the limits of what an LLM is capable of. You don't know that. This is literally just an assertion. An unfounded one at that. If you couldn't predict how far in 2017 the LLM concept would take us today, then you definitely have no idea how far it could actually go. >believes we are approaching the limits of LLM size for size’s sake Nothing to do with thi…
As for as what you linked, Altman is saying the same thing I'm saying:
> That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude.
This is exactly my point; doubling or tripling of the size will be possible, but it won't result in a doubling of performance. We won't see a GPT 5 that's twice as good as GPT 4, for example. The jump from 2 to 3 was exponential. The jump from 3 to 4 was also exponential, though not as much. The jump from 4 to 5 will follow that curve, according to Altman, which means exactly what he said in my quote; the value will continue to decrease. For a 2 to 3 type jump, GPU technology would have to completely transform in capability, which there are no indications that we've found that innovation.