> Any advance could be dismissed as "case-specific", couldn't it?
and
> If by "counting" you mean the "counting r's" issue, that's a tokenisation artifact (the models are blind to individual characters): ask it to count in Python and it's fine (ie. reasoning is not the problem).
The problem is a lack of transparency. That's on the AI companies, not me. Everybody can attach a mathematics engine to an LLM and say "Look, it does math now!". The intelligence part hasn't changed though, the model didn't figure out mathematics, didn't get any closer to AGI.
I am also not saying, testing is impossible (making claims unfalsifiable), but you have to acknowledge it's inherently becoming harder to argue either way. Now, if I asked to count to 100 the LLMs deliver. I can even ask to count to 100 in base16. But it falls apart in less common bases like base33. The problem clearly isn't solved fundamentally. For counting in arbitrary bases you need a knowledge transfer and creative adjustment since you may exceed the symbol space. I think, this is also a good indicator because for the same reason there is no training material. Again, this can be addressed algorithmically, so this test is not futureproof.
Regarding character counting, the excuse about tokenisation is a bit questionable in this context, I think, because inherent limitations to the current approach is exactly the matter debated. Do we know, the extent of this particular problem space in the real world?
>What's something concrete that you would judge as a fundamental step forward?
I want reproducibility and transparency how a feature happened. If you threw a bazillion dollars onto Wolfram Alpha 15 years ago, you would also have a very useful tool for most questions. Spending a lot of money to implement edge case solution is not technological progress. That's always been possible. I mean, in the beginning there were thousands of humans curating chatgpt answers. That's not intelligence, just reckless spending, exploitation and borderline fraud. And it shows exactly how much "customer experience" can be delivered just by throwing money around.
> "Is progress efficiency per dollar decreasing" is a different claim to the earlier "is progress slowing?", though.
Well, I guess it is a matter of opinion. In my opinion, a "thermodynamic" perspective is useful. If you hire 100000 more workers to harvest a small field, you get it done quicker, but is it a breakthrough in farming? That is, trivially, if we spent more energy, resources, money we can expect to get more anything. However, those investments are not spent elsewhere, the progress of technological/scientific advancement is slowing down elsewhere. So, going by relative contributions to overall progress things are slowing down when progress is disproportionally low in an resource intense field. After all resources on Earth are limited and there is only so much sunlight refueling our energy supply. If the AI hype isn't panning out, the damage by hardware prices alone will be enormous. You need compute for more than AI and a lot of that is unaffordable right now. AI has to be 10X for every field, or it's gonna be a net global set back. Look at the volatility and social disruption in the US. It's a gamble, not covered by growth, but debt. Putting progression into relation to effort is required IMO.
> Measured hallucination rates have gone down, and don't seem to be an issue for achieving things, as per METR and other results. I don't see hallucination issues in practice as all real work I know about includes the ability to validate.
Are you in tech? Because I think there is a bias related to the above, since agentic AI can compile/run code to validate answers and most problems are documented ad nauseam. In my experience, if you search for very specific information in other sciences, you still frequently get confident non-sense. Outside of IT, you can't really have a "fact checking" co-routine, you need actual cognition. Of course, if you "chain AIs" things get a little better, but I claim again that's multiplying effort, not fundamentally making anything smarter. If we compare to biological intelligence, we absolutely do not see the same relationship. You may think this argument is moving the goal post, but if I ask you this: If we double the AI workload (eg. agentic AI) to get 10% better results, doesn't this sound like approaching a plateau to you? We are literally putting the fabric of civilization and humanity's future at risk for this (by social disruption, increasing energy/resource expenditure and not spending the money elsewhere in most critical times). Expectations must be high.
Anyway, thank you for expanding the argument, nice discussion.