Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

Post reply on HN
Print view View original

231–240 of 469 posts

Re: “Car Wash” test with 53 models

#231

Post by xyproto »

vasco wrote: post #221
siva7 wrote: post #219

This is probably the greatest one-time AI "Benchmark" ever made. The foundation companies have been gaming traditional benchmarks for years so that no one can really match those numbers into real-world experience. Car wash test tells me on the other hand what kind of intelligence i can expect.

For me it's interesting because no normal person I know would ever inject "because its better for the environment" in anything so small scale so not only it shows they suck, it shows how easy it is to inject side-ideology into simple exchanges.

Referring to "the normal people you know" is purely anecdotal evidence and can't be used to infer anything at all about "side-ideology". Perhaps you only know people that don't care about the environment?

Re: “Car Wash” test with 53 models

#232

Post by krisoft »

Drupon wrote: post #211
denzil wrote: post #207

Earlier quoted context omitted.

Unlike the car question, where you can assume the car is at home and so the most probable answer is to drive, with the machines it gets complicated. Since the question doesn't specify if each machine makes one part or if they depend on each other (which is pretty common for parts production). If they are in series and the time to first part is different than time to produce 5 parts, the answer for 100 machines would…

You passed the intelligence check and failed the wisdom one. The key technique in the mathematical method to answer the machine question is "theory of mind".

Theory of mind won’t help you answering this question. It is obviously an underspecified question (at least in any contexts where you are not actively designing/thinking about some specific industrial process). As such theory of mind indicates that the person asking you is either not aware that they are asking an underspecified question, or are out to get you with a trick. In the first case it is better to ask clarifying question. In the second case your choosen answer depend on your temperament. You can play along with them, or answer an intentionally ridiculous answer, or just kick them in the shin to stop them messing with you.

There is nothing “mathematical” about any of this though.

Re: “Car Wash” test with 53 models

#233

Post by killerstorm »

umairnadeem123 wrote: post #161

[dead]

I don't see anything concerning. Mechanistic interpretability research indicates that LLM internals are inherently parallel: many features "light up" in parallel, then strongest ones "win" and contribute to the output.

I'd guess it suggests walking if a feature indicates that the question is so simple it doesn't warrant step-by-step analysis.

Re: “Car Wash” test with 53 models

#235

Post by vasco »

xyproto wrote: post #231
vasco wrote: post #221

Earlier quoted context omitted.

For me it's interesting because no normal person I know would ever inject "because its better for the environment" in anything so small scale so not only it shows they suck, it shows how easy it is to inject side-ideology into simple exchanges.

Referring to "the normal people you know" is purely anecdotal evidence and can't be used to infer anything at all about "side-ideology". Perhaps you only know people that don't care about the environment?

Majority of people I know care about the environment but they would never inject a phrase like that in a quick exchange about going to wash the car 50m away is my point. In wanting to be a pure heart you missed the actual point.

Re: “Car Wash” test with 53 models

#236

Post by krisoft »

rmunn wrote: post #174
CobrastanJorji wrote: post #164

Earlier quoted context omitted.

I don't think 30% of people can't reason. I think 30% of people will fail fairly simple trick questions on any given attempt. That's not at all the same thing. Some people love riddles and will really concentrate on them and chew them over. Some people are quickly burning through questions and just won't bother thinking it through. "Gotta go to a place, but it's 50 feet away? Walk. Next question, please." Those same…

This. The following question is likely to fool a lot of people, too. "I have a rooster named Pat. (Lots of other details so you're likely to forget Pat is a rooster, not a hen). Pat flies to the top of the roof and lays an egg right on the ridge of the roof. Which way will the egg roll?" But if you omit the details designed to confuse people, they're far less likely to get it wrong: "I have a rooster named Pat. Pat f…

What does “get it wrong” mean for you with this question? Or what is “getting it right” here? If i hear that Pat is a rooster and i understand and retain that information I will look at you like you are dumb for saying such an impossible story. If i don’t i will look at you like you are dumb because how is anyone supposed to know which way will an egg laid on a ridge roll. How are you supposed to even score this?

Re: “Car Wash” test with 53 models

#238

Post by aurareturn »

dozerly wrote: post #131
Niko901ch wrote: post #129

The interesting thing about the 71.5% human baseline is that it suggests the question is more ambiguous than the article claims. When someone asks 'should I walk or drive to the car wash,' a reasonable interpretation is 'should I bother driving such a short distance.' Nearly 30% of humans missing it undermines the framing as a pure reasoning failure - it is partly a pragmatics problem about how we interpret underspec…

I don’t think it’s under specified. You are clearly stating “I want to wash my car”, then asking how you should get there. It’s an easy logical step to know that, in this context, you need your car with you to wash it, and so no matter the distance you should drive. You can ask the human race the simplest, most logical question ever, and a percentage of them will get it wrong.

1. When do you want to wash your car? Tomorrow? Next year? In 50 years?

2. Where is the car now? Is it already at the car wash waiting for you to arrive?

I can see why an LLM might miss this. I think any good software engineer would ask clarifying questions before giving an answer.

The next step for an LLM is to either ask questions before giving a definitive answer for uncertain things or to provide multiple answers addressing the uncertainty.

Re: “Car Wash” test with 53 models

#239

Post by [deleted] »

pcwelder wrote: post #143

To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification im…

[deleted]

Re: “Car Wash” test with 53 models

#240

Post by magackame »

user_7832 wrote: post #154
Nition wrote: post #126

Earlier quoted context omitted.

It highlights a general problem with LLMs, that they always jump to answering, whereas humans will often ask clarifying questions first.

I wonder if anyone has any research on this field. I've often seen this myself ( too often) where LLMs make assumptions and run off with the wrong thing. "This is how you do " or "This is why is impossible!". Ffs man, just ask for info! A human wouldn't need to - they'd get the context - but LLMs apparently don't?

Don't people do this too all the time?
Post reply on HN