> How much do these models need to do before people throw their hands in the air and say, ok this is happening
What is "this"? Most people arguing against some of the more fervent predictions and promises of "inevitability" are people who are using these models in day to day - they see what the models can do, and what they struggle at.
> Now if you have millions of instances running in parallel, all "probabilistic", working on frontier AI research I really don't see the blocker (and believe me I wish I did).
My genuine prediction is that you'll get a lot of early results simply because you're applying attention to some low hanging fruit of problems, but then it will drop off due to the cost of tokens and the low rate of return. This doesn't mean that the models are especially capable of novel thought, just that we haven't algorithmically brute forced a problem with known solutions.
We would be seeing more success cases if the promises were true, setting aside AGI, human replacement, etc. We would see more, better products with more features that people would use. We wouldn't be having any arguments. The human replacement presupposes the models work in ways that they don't, and until proven otherwise, can't. I've watched those who embrace it fully flounder around on projects, some have lost their mind from the constant LLM validation, and I've seen companies go all in and then pull back based on both cost and efficacy over the last year.
I'm still waiting for the success case examples applied on a scale that would make any of the predictions come true.