This is a not-unexpected result if you think of AI as what it actually is instead of what a multi-trillion dollar marketing campaign wants it to be.
At heart, the corpus for this going to be an aggregation of commentary from people in the undisputed most obese era in all of human history performatively denouncing and mocking an imagined other for using cars to go short distances and advocating for walking.
So you've got all "50 meters away? Of course you should walk!" vs a much, much smaller sliver of content about trick questions.
There is no reasoning here, there has never been any reasoning, there has been reasonable or less reasonable weighting for existing reasoning people already did that became part of training data.
If you take away the input corpus, you also take away the illusion of reasoning.
Whereas with other things that can reason like corvids, or ants or octopodes or slime molds, they can derive novel solutions and do a bit of math without any answer key. Mathematics is pure reasoning without any interference and AI can't do it at all unless you provide it with a corpus of already accurate formulas.
> People kept saying humans would fail this too, so I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that.
This really is a grasping at straws ad hoc rationalization for the outcome that is never going to die, and you can see the top comments are efforts to salvage it or cast doubt on the outcome.
If you work for or own a lot of stock in an AI company, I understand you can't understand what you're being paid not to understand. But if you're anyone else...