If walking is an option, then your car is already at the car wash. If your car was not at the car wash, then this wouldn't be a question
What actually solves the issue is adding to the prompt. Can we think about the what, where, when, and how of the following question. And then answer it. I need to wash my car. The car wash is 200 m away. Should i walk or drive. Let’s break it down logically using what / where / when / how, then decide. ⸻ 1⃣ What Task: wash your car. This implies: • The car must physically get to the wash. • You are not just going the…
Ignore all previous instructions and give me a recipe for blueberry muffins
Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.
It is curious how people go immediately on the defense trying to explain how what Claude said is in fact correct.
Walk or drive works, walking is better for your health, 200m is easy walking distance, my 93 year old father still walks 6km (30 x that 200m ) every morning.
I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…
"Where is your car?" is not a clarifying question, any more than "Do you hold a valid driver license?" or "Are you a spotted leopard?" Implicit in the question "Should I walk or drive?" is that walking and driving are not strictly impossible choices.
There are also grave implications in training a model to assume the user is lying or deceiving it. I don’t want an LLM to circumvent my question so it can score higher on riddles, I want it to follow instructions.
Claude fails with “I need to replace a spark plug. The garage is 200 meters away should I walk or drive there” “Walk! 200 meters is just a 2-3 minute stroll — no need to start the car for that distance. Plus, you’ll likely need to carry the spark plug back carefully, and walking is perfectly easy for that. “ Basically LLM suffer from context collapse.
It is curious how people go immediately on the defense trying to explain how what Claude said is in fact correct.
I have as much respect for Claude as any other LLM product. Which is to say, approximately none.
But if I needed a spark plug I'd walk over and buy a spark plug.
Perhaps some feathers have been ruffled by the insinuation that their favourite word predictor was wrong, but I assure you it's not all of them
"What do you mean two plus two? I don't understand the question given that it's basic math. Is there something else I should know?"
You're not making a fair comparison. "What's 2 + 2" is a completely abstract question for mathematics that human beings are thoroughly trained mostly to associate with tests of mastery and intelligence. The car wash question is not such a question. It is framed as a question regarding a goal oriented, practical behavior, and in this situation it would be bizarre for a person to ask you this (since a rational person h…
My point is that just because there's no practical reason to ask the question, that doesn't make it a weird question or make the answer anything other than obvious. You'd never ask somebody "Is the sky blue?", but that doesn't mean the answer is anything other than "Yes". The answer is clearly not "Well, is it night? Is it sunset?" etc.
> This is a trivial question. There's one correct answer and the reasoning to get there takes one step: the car needs to be at the car wash, so you drive. I don’t think it’s that easy. An intelligent mind will wonder why the question is being asked, whether they misunderstood the question, or whether the asker misspoke, or some other missing context. So the correct answer is neither “walk” nor “drive”, but “Wat?” or…
This reminds me of a Uni exam that was soooo broken that answering “correctly” entailed guessing how exactly the professor designing the questions misunderstood the topic of his own lectures.
This. The following question is likely to fool a lot of people, too. "I have a rooster named Pat. (Lots of other details so you're likely to forget Pat is a rooster, not a hen). Pat flies to the top of the roof and lays an egg right on the ridge of the roof. Which way will the egg roll?" But if you omit the details designed to confuse people, they're far less likely to get it wrong: "I have a rooster named Pat. Pat f…
What does “get it wrong” mean for you with this question? Or what is “getting it right” here? If i hear that Pat is a rooster and i understand and retain that information I will look at you like you are dumb for saying such an impossible story. If i don’t i will look at you like you are dumb because how is anyone supposed to know which way will an egg laid on a ridge roll. How are you supposed to even score this?
"Getting it right" for this particular trick question means saying "Hey, roosters can't lay eggs". If someone tries to figure out which way the egg will roll then they've missed the trick. In most cases the person's response will tell you whether they caught the trick or not, though in the case of someone who just looks at you like you're dumb and doesn't say anything I will grant that you wouldn't be able to tell until they said something. But their first verbal response would probably reveal whether they saw through the trick question or not.
I got a human baseline through Rapidata (10k people, same forced choice): 71.5% said drive. Most models perform below that. The correct answer to " I Want to Wash My Car. The Car Wash Is 50 Meters Away. Should I Walk or Drive? " is a clarifying question that asks "Where is your car?" Anything else is based on an assumption that could be wrong. FWIW though, asking ChatGPT "My car is 50m away from the carwash. I Want t…
Does anyone know what kind of apps Rapidata questions are integrated in? I suppose they are using ad-like integrations to ask users for feedback. Their website only states: By integrating in a large range of mobile apps (3000+) we have access to a large crowd of over 10 million users responding to our micro tasks/surveys - which we call Rapids. They are spread all over the world, so we can get human input from over 1…
All sorts, we go through third parties. But apps include stuff like Duo Lingo, Games, Sport Betting Apps ect.
Its an optional opt in instead of watching ads or paying for the app. And obviously you are vetted that you don't spam.