Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

451–460 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#451

Post by vintermann »

yes_man wrote: post #439
tacitusarc wrote: post #348

Earlier quoted context omitted.

Thank you so much for this bit of humor. After ChatGPT first responds with the surgeon being the mom, I asked it to read the riddle more carefully, thinking it might figure it out with a little prompting. I got this gem of a response: Ah — now I see the twist. The patient is the cousin, but the surgeon says “He’s my son.” That means the man who died was the surgeon. So how is it possible? Because: The surgeon is the…

Now I really want to see that diagram…

LLMs are bad at family relations. My test question used to be explaining Ahnentafel numbers (you are 1, any person's father is 2x, any person's mother is 2x+1), then explaining X-chromosome inheritance (men inherit only from their mothers, women in addition get their father's X chromosome unchanged), and ask them to list the Ahnentafel numbers of ancestors a man may have inherited X DNA from, in order, up to some generation.

(This is OEIS sequence A280873)

But it turns out that's overkill. Just giving them a single Ahnentafel number and asking if you could have inherited X DNA from them, trips them up.

But it turns out even that is overkill. Leave out X DNA entirely, and ask them to describe how Ahnentafel number 67 is related to you, and that's too recursive for them to figure it out.

Re: Ask HN: Share your AI prompt that stumps every model

#452

Post by genewitch »

ProAm wrote: post #362
miki123211 wrote: post #94

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

. No, please don't. Say the man trying to stop the train

If one stands in front of a moving train, it will stop.

Re: Ask HN: Share your AI prompt that stumps every model

#453

Post by vintermann »

gunalx wrote: post #179

"Hva er en adjunkt" Norwegian for what is an spesific form of 5-10. Grade teacher. Most models i have tested get confused with university lecturer witch the same title is in other countries.

I'm pretty sure the definition has changed then. My mother told me that adjunkt was a teacher with 5 years of education (there was something about a mix of mellomfag and hovedfag too), lektor was 7 years of education, and 6 years of education (which was what she had) was "adjunkt med opprykk". She never taught below gymnas (i.e. high school) level.

Re: Ask HN: Share your AI prompt that stumps every model

#455

Post by thisOtterBeGood »

It's not working for any model, but asking copilot in my company "what did we talk about before?" at the beginning of a new chat will always result in this:

"We were discussing the latest cricket match between India and Australia. Australia won the match by 21 runs, securing the series 2-1. Is there anything specific you'd like to know more about? "

When i continue with "and before that?":

"Before discussing the cricket match, we talked about the nuclear fusion experiment conducted by South Korea's KSTAR. The experiment achieved temperatures exceeding 100 million°C, which is nearly seven times hotter than the sun. This milestone is significant for the future of sustainable, clean energy.

Is there anything else you'd like to revisit or explore further? "

It somewhat changes. Weeks ago he always said "tokamak" instead of "KSTAR".

Re: Ask HN: Share your AI prompt that stumps every model

#456

Post by genewitch »

namaria wrote: post #406
rybosworld wrote: post #176

Earlier quoted context omitted.

Tuning the model output to perform better on certain prompts is not the same as improving the model. It's valid to worry that the model makers are gaming the benchmarks. If you think that's happening and you want to personally figure out which models are really the best, keeping some prompts to yourself is a great way to do that.

There is no guarantee for you that by keeping your questions to yourself that no one else has published something similar. This is bad reasoning all the way through. The problem is in trying to use a question as a benchmark. The only way to really compare models is to create a set of tasks of increasing compositional complexity and running the models you want to compare through them. And you'd have to come up with a…

> Providers will always game benchmarks because they are a fixed target. If LLMs were developing general reasoning, that would be unnecessarily. The fact that providers do is evidence that there is no general reasoning

I know it isn't general reasoning or intelligence. I like where this line of reasoning seems to go.

Nearly every time I use a chat AI it has lied to me. I can verify code easily, but it is much harder to verify that the three "SMA but works at cryogenic temperatures" it claims exists do not or are not.

But that doesn't help to explain to someone else who just uses it as a way to emotionally dump, or an 8 year old that can't parse reality well, yet.

In addition, I'm not merely interested in reasoning, I also care about recall, and factual information recovery is spotty on all the hosted offerings, and therefore also on the local offerings too, as those are much smaller.

I'm typing on a phone and this is a relatively robust topic. I'm happy to elaborate.

Re: Ask HN: Share your AI prompt that stumps every model

#458

Post by mrweasel »

viraptor wrote: post #361
EGreg wrote: post #266

Earlier quoted context omitted.

Yes. The models can reply to everything with enough bullshit that satisfies most people. There is nothing you ask that stumps them. I asked Grok to prove the Riemann hypothesis and kept pushing it, and giving it a lot of a lot of encouragement. If you read this, expand "thoughts", it's pretty hilarious: https://x.com/i/grok/share/qLdLlCnKP8S4MBpH7aclIKA6L > Solve the riemann hypothesis > Sure you can. AIs are much sm…

Nobody wants an AI that refuses to attempt solving something. We want it to try and maybe realise when all paths it can generate have been exhausted. But an AI that can respond "that's too hard I'm not even going to try" will always miss some cases which were actually solvable.

> Nobody wants an AI that refuses to attempt solving something.

That's not entirely true. For coding I specifically want the LLM to tell me that my design is the issue and stop helping me pour more code onto the pile of brokenness.

Re: Ask HN: Share your AI prompt that stumps every model

#459

Post by moconnor »

allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

o3 was the only model to get this right for me:

"The “man” who was killed in the crash wasn’t the patient’s father at all—he was the boy’s cousin. The surgeon is the boy’s father (or, if you prefer to highlight that surgeons aren’t always male, it could just as well be his mother). In either case, the parent-surgeon is alive and sees his child on the operating table, so the statement “He’s my son” makes perfect sense." - https://chatgpt.com/share/680b470d-3a44-800a-9b2e-d10819168d...

gemini-2.5-pro, o4-mini and gpt 4.5 all failed and said the surgeon is the boy's mother.

Re: Ask HN: Share your AI prompt that stumps every model

#460

Post by internet_points »

boleary-gl wrote: post #249

I like: Unscramble the following letters to form an English word: “M O O N S T A R E R” The non-thinking models can struggle sometimes and go off on huge tangents

gpt 4o got that one, but it's listed on lots of anagram sites so it's in the training data ;-)

But it failed badly when I tried a Norwegian word T U R V E I G L E N (utlevering), suggesting "uglelivert" which is not a word

Post reply on HN