Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

351–360 of 469 posts

Re: “Car Wash” test with 53 models

#351
post #347
post #285

Earlier quoted context omitted.

I also don't trust the maxbenched results. I am thus making my own benchmarks: https://aibenchy.com

In your benchmark, GPT 5 Nano is basically tied with Opus?

Yes. Opus could do a lot better, but fails a lot because it doesn't respect the given formatting instructions/output format.

I could modify the tests to emphasize the requirements, but then, what's the point of a test. In real life, we expect the AI to do something if we ask it, especially for agentic use-case or in n8n, because if the output is slightly wrong, the entire workflow fails.

Re: “Car Wash” test with 53 models

#352

Did AI write the post? First section says "The models that passed the car wash test: ...Gemini 2.0 Flash Lite..." A section or 2 down it says: "Single-Run Results by Model Family: Gemini 3 models nailed it, all 2.x failed" In the section below that about 10 runs it says: 10/10 — The Only Reliable AI Models ... Gemini 2.0 Flash Lite ..." So which it is? Gemini 2.x failed (2nd section) or it succeeded (1st and 3rd) sec…

Flash lite succeeded in every test, smth got lost in editing, just updated it. thx!

Re: “Car Wash” test with 53 models

#353
post #347
post #285

Earlier quoted context omitted.

I also don't trust the maxbenched results. I am thus making my own benchmarks: https://aibenchy.com

In your benchmark, GPT 5 Nano is basically tied with Opus?

Also, not really tied, Opus has a lot better consistency and reasoning score (which means the reasoning made sense, only the final output was wrong).

Re: “Car Wash” test with 53 models

#354

To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification im…

Sonnet 4.6 wasn't part of the test in my case but would be interesting to see the baseline responses. It might be that it gets it right regardless, but will have to test it.

From some rudimentary tests I just did, Sonnet 4.6 says walk consistently. Opus 4.6 days drive pretty consistently.

Re: “Car Wash” test with 53 models

#355

To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification im…

Fundamentally the failure here is one of reasoning/planning - either of not reasoning about the implicit requirements (in this case extremely obvious - in order to wash my car at the car wash, my car needs to be at the car wash) to directly arrive at the right answer, and/or of not analyzing the consequences of any considered answer before offering it as the answer.

While this is a toy problem, chosen to trick LLMs given their pattern matching nature, it is still indicative of their real world failure modes. Try asking an LLM for advice in tackling a tough problem (e.g. bespoke software design), and you'll often get answers whose consequences have not been thought through.

In a way the failures on this problem, even notwithstanding the nature of LLMs, are a bit surprising given that this type of problem statement kinda screams out (at least to a human) that it is a logic test, but most of the LLMs still can't help themselves and just trigger off the "50m drive vs walk" aspect. It reminds a bit of the "farmer crossing the river by boat in fewest trips" type problem that used to be popular for testing LLMs, where a common failure was to generate a response that matched the pattern of ones it had seen during training (first cross with A and B, then return with X, etc), but the semantics were lacking because of failure to analyze the consequences of what it was suggesting (and/or of planning better in the first place).

Re: “Car Wash” test with 53 models

#356

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they c…

Yea funny coincidence, but this is not at all how the human answers were collected.

Rapidata answered this in another comment below. They integrate micro-surveys into mobile apps (like Duolingo, games, etc) as an optional opt-in instead of watching ads. The users are vetted and there's no incentive to answer correctly.

Re: “Car Wash” test with 53 models

#357

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

I don't think this is quite right. It's not that the question is inherently underspecified, it's that the context of being asked a question is itself information that we use to help answer the question. If someone asks "should I walk or drive" to do X, we assume that this is a question that a real human being would have about an actual situation, so even if all available information provided indicates that driving is…

> “…we assume the cooperative principle holds […] that the question is sincere and that there must be some plausible reason for walking.”

Yes. And. Some problems have irrelevant data which should be ignored.

The walk choice is irrelevant in the context. It needs to be simplified, as with a maths problem. That has nothing to do with human nature, but rather prior mistake in reasoning.

Re: “Car Wash” test with 53 models

#358
Maybe relevant to this is that today Dario Amodei is meeting with Pete Hegseth in what Hegseth is describing as a "shit or get off the pot" meeting, with one of the issues being that Hegseth is unhappy with Amodei's unwillingness to have Anthropic models used to make autonomous (no human in the loop) life or death decisions.

Maybe Hegseth should be reading this thread, and/or doing a little reading up on paperclip production maximization.

Re: “Car Wash” test with 53 models

#359

I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…

Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.

Maybe that is because i am non-native speaker, but 'garage' i understand primarily as a place where car is parked, not a car repair shop. So it makes perfect sense to walk there in order to repair the car (that is already there).
Post reply on HN