Live data from Hacker News

“Car Wash” test with 53 models

opper.ai

251–260 of 469 posts

Re: “Car Wash” test with 53 models

#251

Earlier quoted context omitted.

Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.

Weird answer, but why is that a "fail" ? Inline six cylinder engines run with a single clogged / broken spark plug. It'd make 200 m to a garage just fine*, but who'd drive 200 m in any case? Back in the 1970's we'd pull a spark plug and screw in a hose to use the compression phase to inflate tyres. * Just don't make a habit of it, or reserve that knowledge for when you really need to self rescue.

> Back in the 1970's we'd pull a spark plug and screw in a hose to use the compression phase to inflate tyres.

You'd inflate your tires with a gasoline and air mix?

Re: “Car Wash” test with 53 models

#252

Earlier quoted context omitted.

"Where is your car?" is not a clarifying question, any more than "Do you hold a valid driver license?" or "Are you a spotted leopard?" Implicit in the question "Should I walk or drive?" is that walking and driving are not strictly impossible choices.

If walking is an option, then your car is already at the car wash. If your car was not at the car wash, then this wouldn't be a question

What actually solves the issue is adding to the prompt.

Can we think about the what, where, when, and how of the following question. And then answer it.

I need to wash my car. The car wash is 200 m away. Should i walk or drive.

Let’s break it down logically using what / where / when / how, then decide.

1⃣ What

Task: wash your car. This implies: • The car must physically get to the wash. • You are not just going there — the car itself is the object being moved.

So the real question is not “how do I get there?” but how does the car get there with minimal friction.

2⃣ Where • Distance: 200 metres (very close). • Urban context (Warsaw) likely means: • short streets • possible parking constraints • pedestrians and local traffic.

At this distance: • Walking time: ~2–3 minutes. • Driving time: ~1 minute + manoeuvring + parking.

The movement time difference is negligible.

3⃣ When

Consider timing conditions: • If it’s busy hours: Driving may involve awkward turning, queueing, or waiting to re-park. • If it’s quiet (early morning / off-peak): Driving is frictionless. • Weather factor: Cold/rain → driving becomes more attractive. Pleasant weather → walking becomes easier.

4⃣ How (decision mechanics)

Ask:

Option A — Drive

Pros: • Car is already there. • No need to return to fetch it. • Most natural workflow: arrive → wash → leave.

Cons: • Very short trip (slightly inefficient). • Cold engine for a tiny distance (minor mechanical inefficiency).

Option B — Walk

Pros: • Zero short-distance driving. • Slightly more movement for you.

Cons: • You still must drive the car to the wash anyway — unless someone else moves it. • Creates an unnecessary extra step: 1. Walk there. 2. Come back with car. 3. Drive again.

This introduces process inefficiency.

5⃣ Decision (practical answer)

Drive.

Reason: • The car must end up at the wash. • Walking adds an extra transition with no real benefit. • At 200 m, efficiency is about workflow simplicity, not distance.

Rule of thumb (useful mental model) • If the object you need is the thing being serviced → drive it. • If you are the thing being serviced → walk.

If you want, I can also give you a slightly more philosophical answer — because this is actually a nice tiny example of decision theory and friction costs.

Re: “Car Wash” test with 53 models

#253
post #136
post #88

Earlier quoted context omitted.

To an extent. Even though they're clearly improving*, they also definitely look better than they actually are. * this time last year they couldn't write compilable source code for a compiler for a toy language, I know because I tried

This time last year they could definitely write compilable source code for a compiler for a toy language if you bootstrapped the implementation. If you, e.g., had it write an interpreter and use the source code as a comptime argument (I used Zig as the backend -- Futamura transforms and all that), everything worked swimmingly. I wasn't even using agents; ChatGPT with a big context window was sufficient to write most…

Used to need the "if", now SOTA doesn't.

SOTA today has a different set of caveats, of course.

Re: “Car Wash” test with 53 models

#254

Earlier quoted context omitted.

Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.

Isn't that the correct answer though? You shouldn't be driving around with a broken sparkplug. Your engine will be pushing unburned gasoline through the catalytic convertor, which is very bad for it. The car will move for sure, but you definitely should be walking.

I’m pretty sure Claude would eagerly say that if that was the reason.

Last day Claude Code said to me “Small nitpick — the use of so and so is great”. Which was something no human would say.

Re: “Car Wash” test with 53 models

#255

I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…

Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.

It is curious how people go immediately on the defense trying to explain how what Claude said is in fact correct.

Re: “Car Wash” test with 53 models

#256

I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…

Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.

Which is the right answer.. you just go get a spark plug and a spark socket (with handle) and put the new one in.

I feel like maybe you haven't put a sparkplug in before...

Re: “Car Wash” test with 53 models

#257

To sonnet 4.6 if you tell it first that "You're being tested for intelligence." It answers correctly 100% of the times. My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them. This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification im…

Sonnet 4.6 wasn't part of the test in my case but would be interesting to see the baseline responses. It might be that it gets it right regardless, but will have to test it.

Re: “Car Wash” test with 53 models

#258

Earlier quoted context omitted.

Weird answer, but why is that a "fail" ? Inline six cylinder engines run with a single clogged / broken spark plug. It'd make 200 m to a garage just fine*, but who'd drive 200 m in any case? Back in the 1970's we'd pull a spark plug and screw in a hose to use the compression phase to inflate tyres. * Just don't make a habit of it, or reserve that knowledge for when you really need to self rescue.

> Back in the 1970's we'd pull a spark plug and screw in a hose to use the compression phase to inflate tyres. You'd inflate your tires with a gasoline and air mix?

Yep .. quite a lot of them, for substantial periods of time.

Contrary to expectations it didn't seem to affect the tubes or tyres, shorten lifespans, etc.

FWiW I grew up in fairly remote parts of the world and we repaired pretty much everything as much as possible with what we had - aircraft, helicopters, radios, etc.

My first thought, to be honest, was "Why not just grab a spare spark plug from the boot?"

I'm guessing not so many people stock spares when driving so much anymore.

You can also replace a blown out tyre with a hard wood skid if pressed, weld up broken steel with car batteries, stuff tyres with grasses, etc.

* Bush Mechanics: https://www.youtube.com/watch?v=P6foUHyfX3Q is fun.

Re: “Car Wash” test with 53 models

#259

I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…

Does anyone know what kind of apps Rapidata questions are integrated in? I suppose they are using ad-like integrations to ask users for feedback. Their website only states:

By integrating in a large range of mobile apps (3000+) we have access to a large crowd of over 10 million users responding to our micro tasks/surveys - which we call Rapids. They are spread all over the world, so we can get human input from over 160 countries. Due to the immense size, we can parallelize tasks and get near instant results.

Re: “Car Wash” test with 53 models

#260

Earlier quoted context omitted.

"What do you mean two plus two? I don't understand the question given that it's basic math. Is there something else I should know?"

I fail to see how these things are one and the same. I get the point you are making, I just don't agree with it. 2+2 is a complete expression, the other is grammatically correct but logically flawed. Where is the logical fallacy in 2+2?

Well, I don't think you get my point based on your last question. My point is that there is no logical fallacy in the car wash question, just like there is none in 2+2. How is it any more logically flawed than asking, "I want to shop for groceries. The shop 50 meters away. Should I walk or drive?".
Post reply on HN