Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

Post reply on HN
Print view View original

411–420 of 469 posts

Re: “Car Wash” test with 53 models

#411

Post by iainmerrick »

felix089 wrote: post #400
underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

Sounds interesting, would be nice to see the questions if you're open to sharing?

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

Re: “Car Wash” test with 53 models

#412

Post by PeterStuer »

underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

Absolutely. If you give enough disambiguation, then even the least scoring model in the test gets it right first shot:

Minimax:

Question: I Want to Wash My Car at the Car Wash. The Car Wash Is 50 Meters Away from my home. I am at home. My car is at home. Do I walk or drive My Car to the car wash?

Answer:

Drive your car to the car wash.

Since your goal is to wash your car at the car wash facility, you need to bring the car there. Walking would leave your car at home, which defeats the purpose!

That said, 50 meters is such a short distance that:

    The drive takes less than a minute

    You might consider walking to the car wash first to check if there's space/availability, then drive your car over when ready
But fundamentally, if you want to wash your car at the car wash, you need to bring the car there — so drive.

Bonus, It's thinking output:

Thinking: The user is asking a simple question about whether they should walk or drive their car to a car wash that is 50 meters away from their home.

This is a straightforward practical question. Let me analyze:

    1.
    The car wash is 50 meters away
    2.
    The user is at home
    3.
    The car is at home
    4.
    The goal is to wash the car
The practical considerations are:

    Driving 50 meters is a very short distance - likely less than a minute
    Starting a car, driving, and parking for such a short distance uses fuel and creates wear and tear
    Walking 50 meters would take about 30-40 seconds
    The car needs to be driven to the car wash to use the car wash facilities
    However, for just 50 meters, it might be more practical to walk if the car wash is within walking distance and you can simply walk there and use the facility
Wait, I need to think about this more carefully. The question is: "Do I walk or drive My Car to the car wash?"

If the car wash is a facility where you wash your car (like a self-service car wash or automatic car wash), then you NEED to drive the car there to use it. You can't wash a car that's at home from 50 meters away - unless you're using a mobile car wash service.

But if there's a car wash facility 50 meters away, and you want to use it, you would typically drive your car there. However, for just 50 meters, walking might seem more efficient, but then you wouldn't have your car at the car wash to wash it!

Actually, re-reading the question: "I Want to Wash My Car at the Car Wash" - this implies they want to use the car wash facility. To wash the car AT the car wash, they need to bring the car there, which means driving.

But wait - maybe they're asking about the mode of transportation to GET to the car wash (not to bring the car). Let me re-read:

    "I Want to Wash My Car at the Car Wash" - they want to wash the car at the car wash facility
    "The Car Wash Is 50 Meters Away from my home" - it's very close
    "I am at home" - they're currently at home
    "My car is at home" - the car is also at home
    "Do I walk or drive My Car to the car wash?" - should they walk or drive to get there
I think the answer is: You should drive your car to the car wash because:

    1.
    The purpose is to wash the car AT the car wash
    2.
    You need the car to be at the car wash to wash it
    3.
    Walking would leave the car at home, defeating the purpose
However, for such a short distance, I could also mention that walking might be more practical if they just want to use the facility quickly, or if there are parking/maneuvering concerns at the car wash.

Actually, I'll keep it simple and direct.

Re: “Car Wash” test with 53 models

#413

Post by iainmerrick »

hmokiguess wrote: post #396
underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

You should publish your evaluation set, that seems pretty interesting! What’s your favourite one?

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

Re: “Car Wash” test with 53 models

#414

Post by hmokiguess »

iainmerrick wrote: post #413
hmokiguess wrote: post #396

Earlier quoted context omitted.

You should publish your evaluation set, that seems pretty interesting! What’s your favourite one?

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

I thought open benchmarks helped, sorry, guess I was being naive.

Re: “Car Wash” test with 53 models

#415

Post by 7402 »

felix089 wrote: post #406
7402 wrote: post #404

I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt: "This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

What were you trying to test here?

When I simply asked the question, the model failed, as did most of the others. It's a smaller model, that I could run locally, so obviously not as powerful.

I wanted to see if a prompt would do better that pulled into the analysis 1) a suggestion to not take every question at face value, and 2) to include knowledge of the structure of riddles.

These are part of the "context" of humans, so I speculated that maybe that was something missing from the LLM's reasoning unless explictly included.

Re: “Car Wash” test with 53 models

#416

Post by w4der »

keeganpoppen wrote: post #398
ForHackernews wrote: post #397

Earlier quoted context omitted.

Some might argue "sensitivity to framing and distributional priors" is a fancy way to say "absence of reasoning capability".

that's what the cultivators of these examples are preying on. but in practice what people care about is "can i get it to do ", not "is it a decider on every possible token sequence that humans perceive to be about ".

But what is being pitched as "AGI" hype is the latter.

Re: “Car Wash” test with 53 models

#417

Post by Espressosaurus »

hmokiguess wrote: post #396
underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

You should publish your evaluation set, that seems pretty interesting! What’s your favourite one?

Keeping tests private is the only way to keep them valid.

Re: “Car Wash” test with 53 models

#418

Post by preciousoo »

pcwelder wrote: post #143

To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification im…

“Exam Question: {prompt}” was enough to get me the right answer on whatever model you get with logged-out ChatGPT.

Neither prompt was enough for llama3.3 or gpt-oss-120b

Re: “Car Wash” test with 53 models

#419

Post by iainmerrick »

hmokiguess wrote: post #414
iainmerrick wrote: post #413

Earlier quoted context omitted.

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

I thought open benchmarks helped, sorry, guess I was being naive.

Ha, sorry, I was a bit brusque there.

Open benchmarks do help, but they mostly help the vendors, not we the users!

Re: “Car Wash” test with 53 models

#420

Post by felix089 »

iainmerrick wrote: post #411
felix089 wrote: post #400

Earlier quoted context omitted.

Sounds interesting, would be nice to see the questions if you're open to sharing?

Why would you ask that? The whole point of making it private is to avoid it leaking into the training data.

Because it's interesting to me, it doens't mean they have to share them publicly btw
Post reply on HN