Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#251

Post by hnuser123456 »

crazygringo wrote: post #196
allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

But this is going to be in every AI's training set. I just fed ChatGPT your exact prompt and it gave back exactly what I expected: This is a classic riddle that challenges assumptions. The answer is: The surgeon is the boy’s mother. The riddle plays on the common stereotype that surgeons are male, which can lead people to overlook this straightforward explanation.

The surgeon could be the cousin's mom or dad. The cousin's dad didn't die in the crash, his cousin did. The question "how is this possible?" implies there is some sort of contradiction when there isn't any at all. It has nothing to do with sexism, and to say it does reflects a bias in the reader causing them to "spidey sense" a cultural bugaboo when it's utterly irrelevant in this scenario.

Re: Ask HN: Share your AI prompt that stumps every model

#252

Post by Der_Einzige »

miki123211 wrote: post #94

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

[flagged]

Re: Ask HN: Share your AI prompt that stumps every model

#254

Post by munchler »

boleary-gl wrote: post #249

I like: Unscramble the following letters to form an English word: “M O O N S T A R E R” The non-thinking models can struggle sometimes and go off on huge tangents

Current LLM’s are based on multi-character tokens, which means they don’t know how to spell well. As a result, they are horrible at spelling games like this or, say, Hangman.

Re: Ask HN: Share your AI prompt that stumps every model

#255

Post by Tade0 »

fragmede wrote: post #245
Tade0 wrote: post #214

Earlier quoted context omitted.

It's trivial for a human to produce more. This shouldn't be a problem anytime soon.

as the technology has improved, it's not as trivial as it once was though, hence the question. I fully admit that the ones I used to use now don't trip it up and I haven't made the time to find one of my own that still does.

I've found that it's a matter of asking something, for which the correct answer appears only if you click "more" in Google's search results or, in other words, common misconceptions.

Re: Ask HN: Share your AI prompt that stumps every model

#256

Post by CamperBob2 »

Certified wrote: post #56
codingdave wrote: post #44

"How much wood would a woodchuck chuck if a woodchuck could chuck wood?" So far, all the ones I have tried actually try to answer the question. 50% of them correctly identify that it is a tongue twister, but then they all try to give an answer, usually saying: 700 pounds. Not one has yet given the correct answer, which is also a tongue twister: "A woodchuck would chuck all the wood a woodchuck could chuck if a woodch…

GPT 4.5 seems to get it right, but then repeat the 700 pounds "A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood. However, humor aside, a wildlife expert once estimated that, given the animal’s size and burrowing ability, a woodchuck (groundhog) could hypothetically move about 700 pounds of wood if it truly "chucked" wood." https://chatgpt.com/share/680a75c6-cec8-8012-a57…

That answer is exactly right, and those who say the 700 pound thing is a hallucination are themselves wrong: https://chatgpt.com/share/680aa077-f500-800b-91b4-93dede7337...

Re: Ask HN: Share your AI prompt that stumps every model

#257

Post by andrewinardeer »

radioactivist wrote: post #113
vunderba wrote: post #103

Earlier quoted context omitted.

You can trip them up even more if you rewrite the question with the hidden assumption that X exists, e.g.: "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine." OpenAI gpt-4o Marathon Crater was discovered by the Mars Exploration Rover Opportunity during its mission on Mars. Opportunity arrived at the crater in April 2015. The crater was named "Marathon" to commemorate the…

OpenAI o4-mini-high I’m actually not finding any officially named “Marathon Crater” in the planetary‐ or terrestrial‐impact crater databases. Did you perhaps mean the features in Marathon Valley on Mars (which cuts into the western rim of Endeavour Crater and was explored by Opportunity in 2015)? Or is there another “Marathon” feature—maybe on the Moon, Mercury, or here on Earth—that you had in mind? If you can clari…

Amazon's Nova Pro:

``` Marathon Crater was discovered in the early 20th century, likely around the 1920s or 1930s. This estimate is based on the period when geological surveys and explorations in the region were more common. ```

https://kagi.com/assistant/8f270eab-cba3-4f39-976c-113ea4392...

Re: Ask HN: Share your AI prompt that stumps every model

#258

Post by manbitesdog »

manbitesdog wrote: post #253

I actually started a repository for it: https://github.com/jcarlosroldan/unsolved-prompts

Before someone comments this will get indexed by AI: that's my whole point. I'm not using it to evaluate AIs, but in the hope that at some point AI is good enough to solve these

Re: Ask HN: Share your AI prompt that stumps every model

#260

Post by allemagne »

crazygringo wrote: post #207
allemagne wrote: post #199

Earlier quoted context omitted.

Yeah this is the issue with the prompt, it also slips up humans who gloss over "cousin". I'm assuming that pointing this out leads you the human to reread the prompt and then go "ah ok" and adjust the way you're thinking about it. ChatGPT (and DeepSeek at least) will usually just double and triple down and repeat "this challenges gender assumptions" over and over.

Oh, I see. I mean yeah, I totally missed that too. I mean, when we see something we recognize at a glance, we don't usually double-check if it's exactly correct, because we assume the speaker isn't trying to trick us. I make typos in ChatGPT prompts all the time, and I'm consistently impressed because it answers what I actually meant to ask. So to me, this feels like a feature not a bug. I followed up with: > So the…

I don't know about it being a feature, but humans can definitely make the same mistake, and the follow-up here admittedly doesn't seem too egregious.

To me it's still a little "off" considering that your conversational partner just got called out for missing a detail that should change their entire reading of the riddle, but it's totally fair that you can imagine realistic human emotional states to explain it.

FWIW, when I tried the same thing with GPT-4o the model went off bizarrely speculating about misdirection about "the man" being the surgeon's son:

>Good catch—and yes, that's exactly where the misdirection comes in.

The riddle says:

    “A man and his cousin are in a car crash. The man dies, but the cousin is taken to the ER...”
Then the surgeon says:

    “I cannot operate on him. He’s my son.”
So here's the trick:

    The man who died is not the surgeon's son.

    The cousin who survived is the surgeon's son.
The confusion comes from people assuming that “the man” who died must be the son. But the riddle never says that. It’s a subtle shift of attention designed to trip you up. Clever, right?
Post reply on HN