Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

371–380 of 469 posts

Re: “Car Wash” test with 53 models

#371

To me the only acceptable answer would be “what do you mean?” or “can you clarify?” if we were to take the question seriously to begin with. People don’t intentionally communicate with riddles and subliminal messages unless they have some hidden agenda.

Sure, if an open ended response was allowed, but if it was a multiple choice question then you'd have to use your common sense and pick one.

However, the important issue here really isn't about the ability of humans or LLMs to recognize logic puzzles. If you were asking an LLM for real world advice, trying to be as straightforward as possible, you may still get a response just as bad as "walk", but not be able to recognize that it was bad, and the reason for the failure would be exactly the same as here - failure to plan and reason through consequences.

It's toy problems like this that should make you step back once in a while and remind yourself of how LLMs are built and how they are therefore going to fail.

Re: “Car Wash” test with 53 models

#372

Earlier quoted context omitted.

I don't think this is quite right. It's not that the question is inherently underspecified, it's that the context of being asked a question is itself information that we use to help answer the question. If someone asks "should I walk or drive" to do X, we assume that this is a question that a real human being would have about an actual situation, so even if all available information provided indicates that driving is…

We could probably test this. I wonder if the results shift if the question is prefaced with something like "Here is a trick question: ...".

I tested both Sonnet and Haiku from Claude, which got it right 0/10 times in their original test, and they both passed. Here's the Haiku output:

"You should *drive*!

The trick is that you need to take your car to the car wash to get it washed. If you walked, your car would still be at home, unclean. So while 50 meters is a short distance that you could walk under normal circumstances, in this case you have to drive because your car is what needs to be washed."

Re: “Car Wash” test with 53 models

#373
post #269

Earlier quoted context omitted.

What does “get it wrong” mean for you with this question? Or what is “getting it right” here? If i hear that Pat is a rooster and i understand and retain that information I will look at you like you are dumb for saying such an impossible story. If i don’t i will look at you like you are dumb because how is anyone supposed to know which way will an egg laid on a ridge roll. How are you supposed to even score this?

"Getting it right" for this particular trick question means saying "Hey, roosters can't lay eggs". If someone tries to figure out which way the egg will roll then they've missed the trick. In most cases the person's response will tell you whether they caught the trick or not, though in the case of someone who just looks at you like you're dumb and doesn't say anything I will grant that you wouldn't be able to tell un…

For me, I would interpret this as being that actually Pat is a hen and the original premise was mistaken. I.e. Pat is not a rooster.

Re: “Car Wash” test with 53 models

#375

This doesn’t look like a reasoning ceiling. It looks like a decision reliability problem. The unstable tier is the key result. Models that get it right 70–80% of the time are not “almost correct.” They are nondeterministic decision functions. In production that’s worse than being consistently wrong. A single sampled output is just a proposal. If you treat it as a final decision, you inherit its variance. If you treat…

Did you write this COMMENT with ChatGPT?!

Come on, man.

Re: “Car Wash” test with 53 models

#376

This doesn’t look like a reasoning ceiling. It looks like a decision reliability problem. The unstable tier is the key result. Models that get it right 70–80% of the time are not “almost correct.” They are nondeterministic decision functions. In production that’s worse than being consistently wrong. A single sampled output is just a proposal. If you treat it as a final decision, you inherit its variance. If you treat…

> This doesn’t look like a reasoning ceiling. It looks like a decision reliability problem.

This doesn’t look like a human comment. It looks like a LLM response.

Re: “Car Wash” test with 53 models

#377
post #195

Earlier quoted context omitted.

People often trip up on similar questions, anything to do with simple math. You know when they go out in the street and ask random people if 5 machines can produce 5 parts in 5 minutes, how long will it take for 100 machines.

There are different kind of statements. Do you mean in a defined time interval or on average? Men are stronger than women. Does that mean there is no woman who is stronger then a man? You can't drive over 50 here. Does that mean it's physically impossible?

Well, these type of questions are looking for intelligent assumptions. Similar to IQ tests, you are supposed to understand patterns and make educated guesses.

Re: “Car Wash” test with 53 models

#378

Interesting that GPT-5.1 and 5.2 (0 of 10 correct) are a lot worse than the older GPT-5 (7 of 10 correct). But unfortunately the article doesn't mention whether they used the reasoning model or not. Even more interesting: Gemini 2.0 Flash Lite got a perfect score (10/10) despite being a quite small and old model.

> But unfortunately the article doesn't mention whether they used the reasoning model or not.

You can run the test yourself if you ask GPT-5.2 with reasoning effort high or xhigh, it will always answer correctly. So if the got 0 from 10, they used zero reasoning efforts which easily explain the results.

Re: “Car Wash” test with 53 models

#379

Interesting that GPT-5.1 and 5.2 (0 of 10 correct) are a lot worse than the older GPT-5 (7 of 10 correct). But unfortunately the article doesn't mention whether they used the reasoning model or not. Even more interesting: Gemini 2.0 Flash Lite got a perfect score (10/10) despite being a quite small and old model.

Good question, I used the API defaults across the board since it felt like the most reasonable baseline to compare. Flash lite getting 10/10 was definitely very surprising

Re: “Car Wash” test with 53 models

#380

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they c…

The default model of ChatGPT is GPT 5.2 Instant, not the one lines up with human results (which is GPT 5).

However, it does tell us something about human answers as the above commenter confidently reached such a strong but baseless conclusion.

Post reply on HN