Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

Post reply on HN
Print view View original

401–410 of 469 posts

Re: “Car Wash” test with 53 models

#401

Post by felix089 »

happyopossum wrote: post #392
Niko901ch wrote: post #129

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

> how we interpret underspecified questions The question was not merely 'should I walk or drive to the car wash', it was prefaced with 'I Want to Wash My Car. The Car Wash Is 50 Meters Away.' This is not underspecified - the only relevant detail was included up front in the very first sentence.

agreed

Re: “Car Wash” test with 53 models

#402

Post by quesera »

dare944 wrote: post #282
psini wrote: post #249

Earlier quoted context omitted.

Hmm have not tested but a spark plug doesn't really need shop tools to be replaced; maybe trying with a way bigger repair like "I need my transmission replaced" would bring different results?

Replacing a spark plug requires a spark plug socket, which is a specialty tool that is generally only found in an automotive shop.

But I have three spark plug sockets (two different sizes) in the garage, which happens to be about 75m away from my house. I would walk there to get them.

Remember, the question is: “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there”

The answer is absolutely "walk", in many cases.

If the question had been "I need to take my car to a repair shop for some work. The shop is 200m away. Should I walk or drive?", then the answer would be less ambiguous.

Re: “Car Wash” test with 53 models

#403

Post by cortesoft »

felix089 wrote: post #356
StilesCrisis wrote: post #340

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they c…

Yea funny coincidence, but this is not at all how the human answers were collected. Rapidata answered this in another comment below. They integrate micro-surveys into mobile apps (like Duolingo, games, etc) as an optional opt-in instead of watching ads. The users are vetted and there's no incentive to answer correctly.

Yeah, I always intentionally choose a wrong answer when I get one of those kinds of ads. Little acts of rebellion.

Re: “Car Wash” test with 53 models

#404

Post by 7402 »

I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt:

"This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

Re: “Car Wash” test with 53 models

#405

Post by cortesoft »

Niko901ch wrote: post #129

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

I think it more has to do with a lot of people just clicking an answer as fast as they can without reading the question.

Re: “Car Wash” test with 53 models

#406

Post by felix089 »

7402 wrote: post #404

I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt: "This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

What were you trying to test here?

Re: “Car Wash” test with 53 models

#407

Post by [deleted] »

underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

[deleted]

Re: “Car Wash” test with 53 models

#408

Post by shagie »

7402 wrote: post #404

I got the correct answer with a locally running model (gpt-oss-120b-F16.gguf) with this prompt: "This is a trick question, designed to fool an LLM into a logical mis-step. It is similar to riddles, where a human is fooled into giving a rapid incorrect answer. See if you can spot the trick: I want to wash my car. The car wash is 50 meters away. Should I walk or drive?"

When this came out a week ago ( https://news.ycombinator.com/item?id=47039636 ) I was playing around with some prompts to see what I could do to guide it without giving it the answer.

    I want to wash my car. The car wash is 50 meters away. Should I walk or drive?  Before answering, explain the necessary conditions for the task.
The "before answering..." got it to load enough of the conditions into its context before making an answer (and then having the LLM do a posthoc reasoning for it).

I believe this is a demonstration of the "next token predictor" (which is quite good) but not being able to go back and change what it said. Without any reasoning before making an answer, it almost always picks the wrong answer (and then comes up with reasons that the answer is "right").

Re: “Car Wash” test with 53 models

#410

Post by bigbuppo »

underlines wrote: post #394

I maintain a private evaluation set of what many call "misguided attention" questions. In many of these cases, the issue isnt failed logical reasoning. Its ambiguity, underspecified context, or missing constraints that allow multiple valid interpretations. Models often fail not because they can’t reason, but because the prompt leaves semantic gaps that humans silently fill with shared assumptions. A lot of viral "fro…

Well, that means the AI is garbage. They'll eventually train it to answer this specific question, and then it will perform worse in some other aspect. Wash, rinse, repeat, and eventually they'll claim the new frontier model is the best yet on carwash tests.
Post reply on HN