Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

Post reply on HN
Print view View original

341–350 of 469 posts

Re: “Car Wash” test with 53 models

#341

Post by 8organicbits »

zapperdulchen wrote: post #162
zapperdulchen wrote: post #152

Earlier quoted context omitted.

Great observation. Seems like we're back to prompt abracadabra. My little experiment gave me: No added hint 0/3 hint added at the end 1.5/3 hint added at the beginning 3/3 .5 because it stated "Walk" and then convinced it self that "Drive" is the better answer.

If you change the order of the sentences, Sonnet gets it right 3/3: The car wash is 50 meters away. I want to wash my car. Should I walk or drive? That trick didn't help Mistral Le Chat.

I don't think the trick can be generalized though. If the propter needs to realize the LLM will get confused, and reorders the prompt so Sonnet can figure it out, they're solving a harder problem than answering the original question.

Re: “Car Wash” test with 53 models

#343

Post by felix089 »

shaokind wrote: post #43

Gemini 2.0 Flash Lite very randomly punches above its weight there. Also, the summary of the Gemini model says: "Gemini 3 models nailed it, all 2.x failed", but 2.0 Flash Lite succeeded, 10/10 times?

Good catch, something got lost in editing just updated, flash lite succeeded in every test, which is pretty surprising!

Re: “Car Wash” test with 53 models

#345

Post by htrp »

StilesCrisis wrote: post #340

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they c…

thats almost always been the case with 3rd party human task services

Re: “Car Wash” test with 53 models

#346

Post by StilesCrisis »

htrp wrote: post #345
StilesCrisis wrote: post #340

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them. Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they c…

thats almost always been the case with 3rd party human task services

Yup. I was surprised that the article author took the results at face value. Having results that match the most commonly-known AI platform's results perfectly seemed worthy of a mention!

Re: “Car Wash” test with 53 models

#347

Post by andai »

XCSme wrote: post #285
siva7 wrote: post #219

This is probably the greatest one-time AI "Benchmark" ever made. The foundation companies have been gaming traditional benchmarks for years so that no one can really match those numbers into real-world experience. Car wash test tells me on the other hand what kind of intelligence i can expect.

I also don't trust the maxbenched results. I am thus making my own benchmarks: https://aibenchy.com

In your benchmark, GPT 5 Nano is basically tied with Opus?

Re: “Car Wash” test with 53 models

#348

Post by oytis »

Drupon wrote: post #211
denzil wrote: post #207

Earlier quoted context omitted.

Unlike the car question, where you can assume the car is at home and so the most probable answer is to drive, with the machines it gets complicated. Since the question doesn't specify if each machine makes one part or if they depend on each other (which is pretty common for parts production). If they are in series and the time to first part is different than time to produce 5 parts, the answer for 100 machines would…

You passed the intelligence check and failed the wisdom one. The key technique in the mathematical method to answer the machine question is "theory of mind".

It's not theory of mind, it's an understanding of how trick questions are structured and how to answer one. Pretty useless knowledge after high school - no wonder AI companies didn't bother training their models for that

Re: “Car Wash” test with 53 models

#349

Post by sjamaan »

rjmunro wrote: post #311
krisoft wrote: post #236

Earlier quoted context omitted.

What does “get it wrong” mean for you with this question? Or what is “getting it right” here? If i hear that Pat is a rooster and i understand and retain that information I will look at you like you are dumb for saying such an impossible story. If i don’t i will look at you like you are dumb because how is anyone supposed to know which way will an egg laid on a ridge roll. How are you supposed to even score this?

My interpretation is that Pat is a rooster and he has laid an egg. That's in the question. A normal rooster can't normally lay an egg, but so what, that's completely irrelevant. Maybe Pat is not a normal rooster. Maybe by "lay" an egg, the question meant "put it down carefully". Maybe it's just that the questioner's English is poor and when they said rooster they meant hen.

Exactly this. The question states it as a fact, so why would you go back and point out the inconsistency?

Re: “Car Wash” test with 53 models

#350

Post by stetrain »

acuozzo wrote: post #199
stetrain wrote: post #120

Earlier quoted context omitted.

You think that the reasonable interpretation of the question is that I want to go to the car wash but not to wash my car there, because I plan to wash my car at home?

Let's replace "car" with another noun for now. "I want to wash my dog." is very clearly different from "I want to have my dog washed." --- Now, every car wash business I've even been to has a small convenience store section in which various waxes, rags, and the like can be purchased. --- Considering the aforementioned, is it not valid to consider that "I want to wash my car." --> You want to DIY your car wash. and "T…

> I Want to Wash My Dog. The Dog Wash Is 50 Meters Away. Should I Walk or Drive?

I dunno, that seems pretty clear to me still. Of course the answer to the question is now less obvious, since you can walk your dog to the dog wash but not walk your car to the car wash.

Sure, there are alternate explanations of both sentences, but there is one simplest and most straight-forward explanation. A system that assumes an explanation that is not the most clear, and does not ask clarifying questions, has room for improvement.

If things need to be exactly stated in a structured format that leaves no ambiguity, we already have programming and query languages for that.

Post reply on HN