Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

Post reply on HN
Print view View original

331–340 of 469 posts

Re: “Car Wash” test with 53 models

#331

Post by 1718627440 »

mk89 wrote: post #208
snmx999 wrote: post #141

Earlier quoted context omitted.

The question does not specify where you or the car are. It specifies only that the car wash is 50 meters away from something, possibly you, the car, or both.

It could also mean there is literally no possible way to reach it, because that's on the other side of a river, and there is no bridge. You should still not "walk there, because come on don't be lazy, a bit of walking is good".

This. To be correct you must also give the answer for the right reason. If you say "drive" but for the wrong reason, then you are still wrong.

Re: “Car Wash” test with 53 models

#332

Post by coldtea »

>so I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive.

What kind of idiot would say "walk"?

I guess if they were average and above they wouldn't gig at Rapidata. Either that or foreigners with mediocre understanding of the English language. Or some bored teenagers that get their $0.20 or whatever Rapidata pays whatever they click (was it a multiple choice) - the (studied and reported) regression in reading comprehension levels doesn't help either.

Re: “Car Wash” test with 53 models

#333

Post by 1718627440 »

nozzlegear wrote: post #30
layer8 wrote: post #14

> This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I don’t think it’s that easy. An intelligent mind will wonder why the question is being asked, whether they misunderstood the question, or whether the asker misspoke, or some other missing context. So the correct answer is neither “walk” nor “drive”, but “Wat?” or…

I think most people would say "drive?" and wonder when the punchline is coming, but (IMO) I don't think they'd start asking for clarification right away.

You can ask with your eyes, tone or face expression. That "?" at the end of "drive?" is a request for clarification or a punchline.

Re: “Car Wash” test with 53 models

#334

Post by 1718627440 »

jiggawatts wrote: post #266
layer8 wrote: post #14

> This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I don’t think it’s that easy. An intelligent mind will wonder why the question is being asked, whether they misunderstood the question, or whether the asker misspoke, or some other missing context. So the correct answer is neither “walk” nor “drive”, but “Wat?” or…

This reminds me of a Uni exam that was soooo broken that answering “correctly” entailed guessing how exactly the professor designing the questions misunderstood the topic of his own lectures.

Are you not allowed to ask the professor questions? We are, and it is not to seldom that the professor then walks to the black board and updates the question.

Re: “Car Wash” test with 53 models

#335

Post by coldtea »

lich_king wrote: post #185
akersten wrote: post #130

Earlier quoted context omitted.

It tracks with the approximate 70:30 split we inexplicably observe in other seemingly unrelated population-wide metrics, which I suppose makes sense if 30% of people simply lack the ability to reason. That seems more correct than me than "the question is framed poorly" - I've seen far more poorly framed ballot referendums.

Is this your experience? Do you think 30% of your friends or family members can't answer this question? If not, do you think your friends or family are all better than the general population? I'd look for explanations elsewhere. This was an online survey done by a company that doesn't specialize in surveys. The results likely include plenty of people who were just messing around, cases of simple miscommunication (e.g…

>If not, do you think your friends or family are all better than the general population?

Since most people live in social bubbles that would be a very plausible case, especially on HN.

If you're a college educated developer, with a college educated wife, and smart, well educated children, perhaps yourselves the children of college educated parents, and your social circle/friends are of similar backgrounds, you'd of course be "better than the general population".

Re: “Car Wash” test with 53 models

#336

Post by 1718627440 »

rapidata wrote: post #73
padjo wrote: post #56

That human baseline is wild. Either the rapid data test is methodologically flawed or the entire premise of the question is invalid and people are much stupider than even I, a famed misanthrope, think.

We were surprise ourselfes, but if you walk around and randomly ask people in the street, I think you would be surprised what you would find. Its a trick question.

Yeah, the point of a trick question is to have fun. So the correct answer is obviously that one that makes the other laugh.

Re: “Car Wash” test with 53 models

#337

Post by oytis »

Niko901ch wrote: post #129

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

Yeah, it's an obvious trick question - as in as a human I read it as such. I think it's a bad benchmark for a model's reasoning ability. If you want to know what the model would do in a real world scenario, you should put this decision in an appropriate context - e.g. when a model should plan one's route for a day using different available means of transportation.

Re: “Car Wash” test with 53 models

#338

Post by felix089 »

Zobat wrote: post #325
Niko901ch wrote: post #129

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

I wonder about the the service used for the test, never heard of Rapidata but if it's like Amazons mechanical turk och other such services there might be a problem where the respondents simply didn't care about reading the question. If the objective for the respondents were simply "answer this question and get your benefit" vs "answer this question correctly to get your benefit" I have no problem accepting the 71.5%…

They answered it in another comment somewhere below, there's no incentive for a correct answer

Re: “Car Wash” test with 53 models

#339

Post by aurareturn »

kklisura wrote: post #278
aurareturn wrote: post #238

Earlier quoted context omitted.

1. When do you want to wash your car? Tomorrow? Next year? In 50 years? 2. Where is the car now? Is it already at the car wash waiting for you to arrive? I can see why an LLM might miss this. I think any good software engineer would ask clarifying questions before giving an answer. The next step for an LLM is to either ask questions before giving a definitive answer for uncertain things or to provide multiple answers…

3. Is the car broken somewhere? Does it have wheels on? 4. Does the car have enough fuel? Jokes asides, all of those questions are unnecessary. There's no more context to this.

If you ask a human that in person, they'd wonder why you'd ask such as stupid question.

I think LLMs should ask clarifying questions if it thinks it's a trick question.

Re: “Car Wash” test with 53 models

#340

Post by StilesCrisis »

What do you know, the human results line up exactly with ChatGPT. What are the odds! Surely the human responders are highly ethical individuals and they wouldn't even dream of copy-pasting all the questions into ChatGPT without reading them.

Realistically, this mostly tells me that the "human answers" service is dead. People will figure out a way to pass the work off to an AI, regardless of quality, as long as they can still get paid.

Post reply on HN