This analogy breaks down when you consider what "attention is all you need" did for us.
A mere 5 years ago I was firmly on the camp (alongside linguists mainly) that believed intelligence requires more than just lots of data and a next token predictor. I was clearly wrong and would have lost a $1000 bet it I had put my money where my mouth was back then. Anyone not noticing how far things have come I think are mostly moving goalposts and falling to hindsight bias.
A better analogy is that the genius person in the village built a step ladder made of carbon nanotubules. Some people proclaimed "All with have to do is keep going and we can reach the moon with a very tall ladder!' Other people - many quite smart - proclaimed reasonably: "This is impossible. You are not realizing the unique challenges and materials we have not yet researched we need to build something like that."
Some in society kept building. The ladder kept getting higher. They run into issues like oxygen and balance so the ladder is redesigned into an elevator. Challenges came and were thought insurmountable until redesigns were still found to work with the miraculous carbon nanotubule material which seemed like a panacea for every construction ill.
Regardless of how high the now elevator gets and regardless of how many times the elevator gets higher than what the naysayers firmly believed is impossible the same naysayers keep saying they will never get much higher.
And higher the elevator grows.
Eventually a limit is reached, but that limit ends up being far higher than any naysayers ever thought possible. And when the limit is reached the naysayers all gathered and said "told you this would be the limit and that it was impossible!'
The naysayers failed to see the carbon nanotubules for the revolutionary potential that it had, even if they were correct that it wasn't enough.
And little did everyone know that their society was mere months ago from another genius being born that would give them another catalyst on the order of carbon nano tubules that would again lead to dramatic unexpected and long-term gains to how high the elevator can grow.