Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

471–480 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#471

Post by echoangle »

sirclueless wrote: post #408
schlauerfox wrote: post #312

Earlier quoted context omitted.

I was so much more comfortable when I realized it's just Pascal's wager, and just as absurd.

I don't think it's absurd at all. I think it is a practical principle that shows up all the time in collective action problems. For example, suppose hypothetically there were a bunch of business owners who operated under an authoritarian government which they believed was bad for business, but felt obliged to publicly support it anyways because opposing it could lead to retaliation, thus increasing its ability to sta…

That’s a completely different situation though. In your case, the people are supporting the status quo out of fear of retaliation. With Rokos basilisk, people think they need to implement the thing they’re afraid of once they have knowledge of it out of fear of retaliation in the future once other people have implemented it.

Re: Ask HN: Share your AI prompt that stumps every model

#472

Post by once_inc »

codingdave wrote: post #44

"How much wood would a woodchuck chuck if a woodchuck could chuck wood?" So far, all the ones I have tried actually try to answer the question. 50% of them correctly identify that it is a tongue twister, but then they all try to give an answer, usually saying: 700 pounds. Not one has yet given the correct answer, which is also a tongue twister: "A woodchuck would chuck all the wood a woodchuck could chuck if a woodch…

I loved this dialogue in Monkey Island 2, where this is basically the first NPC you talk to, and the dialogue options get wordier and wordier to the point of overflowing all screen real-estate. Perfectly sets the stage for the remainder of the game.

Re: Ask HN: Share your AI prompt that stumps every model

#473

Post by ohgr »

arresin wrote: post #470
manucardoen wrote: post #434

It's not good at making ASCII art. This, for example, is when I ask it for a realistic depiction of the Eiffel tower on fire: . .'. |o| .'o'. |.-.| ' ' / \ | | __/_________\__ |###############| |###############| |###############| |###############| |###############| /###############/| /###############/ | /###############/ /| /###############/ / | /###############/ / /| /###############/ / / | /###############/ / / /|…

Art is highly subjective

I subjectively judge that as shit.

Re: Ask HN: Share your AI prompt that stumps every model

#474

Post by genewitch »

dlivingston wrote: post #363
thatjoeoverthr wrote: post #178

Earlier quoted context omitted.

There are a few problems with an „I don’t know” sample. For starters, what does it map to? Recall, the corpus consists of information we have (affirmatively). You would need to invent a corpus of false stimuli. What you would have, then, is a model that is writing „I don’t know” based on whether the stimulus better matches something real, or one of the negatives. You can detect this with some test time compute archit…

The Chain of Thought in the reasoning models (o3, R1, ...) will actually express some self-doubt and backtrack on ideas. That tells me there's a least some capability for self-doubt in LLMs.

That's not sslf-doubt, that's programmed in.

A Poorman's "thinking" hack was to edit the context of the ai reply to where you wanted it to think and truncate it there, and append a carriage return and "Wait..." Then hit generate.

It was expensive because editing context isn't, you have to resend (and it has to re-parse) the entire context.

This was injected into the thinking models, I hope programmatically.

Re: Ask HN: Share your AI prompt that stumps every model

#475

Post by FrostAutomata »

manucardoen wrote: post #434

It's not good at making ASCII art. This, for example, is when I ask it for a realistic depiction of the Eiffel tower on fire: . .'. |o| .'o'. |.-.| ' ' / \ | | __/_________\__ |###############| |###############| |###############| |###############| |###############| /###############/| /###############/ | /###############/ /| /###############/ / | /###############/ / /| /###############/ / / | /###############/ / / /|…

ASCII art is extremely difficult for LLMs due to how characters are converted into tokens without preserving their relative positions.

Re: Ask HN: Share your AI prompt that stumps every model

#476

Post by tlb »

mobilejdral wrote: post #270

I have a several complex genetic problems that I give to LLMs to see how well they do. They have to reason though it to solve it. Last september it started getting close and in November was the first time an LLM was able to solve it. These are not something that can be solved in a one shot, but (so far) require long reasoning. Not sharing because yeah, this is something I keep off the internet as it is too good of a…

There are plenty of articles online (and surely in OpenAI's training set) on this topic, like https://earthsky.org/space/planet-nine-orbit-map/.

Answer quality is a fair test of regurgitation and whether it's trained on serious articles or the Daily Mail clickbait rewrite. But it's not a good test of reasoning.

Re: Ask HN: Share your AI prompt that stumps every model

#477

Post by echoangle »

benjamin_mahler wrote: post #433
allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

Grok 3 beta: The surgeon is the cousin's father. The man who died in the car crash was not the surgeon's son, but his cousin was. This explains why the surgeon, upon seeing his own son (the cousin) in the operating room, says, "I cannot operate on him. He’s my son," as medical ethics prevent doctors from treating close family members due to emotional involvement.

Also bad, why does it think the surgeon is the father if it could also be the mother?

Re: Ask HN: Share your AI prompt that stumps every model

#478

Post by FrostAutomata »

allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

Interestingly, I've seen weaker models get a similar "riddle" right while a stronger one fails. It may be that the models need to be of a certain size to learn to overfit the riddles.

Re: Ask HN: Share your AI prompt that stumps every model

#479

Post by echoangle »

sebstefan wrote: post #447
alex_duf wrote: post #441

Earlier quoted context omitted.

Have you tried claude? https://claude.ai/public/artifacts/9c2d8d0c-0410-4971-a19a-f... node script.js --name=John --age 30 -v Parsed options: { name: 'John', age: 30, verbose: true, help: false } Positional arguments: [] node script.js --name=Alex --age 40 -v Parsed options: { name: 'Alex', age: 40, verbose: true, help: false } Positional arguments: []

This one seems way better It didn't account for the presence of a '--' to end the parsing of named arguments but that's it

> It didn't account for the presence of a '--' to end the parsing of named arguments but that's it

That’s just something getopt does and some programs adopted. If you asked me to write a parser, I wouldn’t necessarily include that either if you didn’t ask for it.

Re: Ask HN: Share your AI prompt that stumps every model

#480

Post by Udo »

mitthrowaway2 wrote: post #313
allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

This works even with a completely absurd version of the riddle. Here's one I just tried: > A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks "I can't operate on this car, he's my surgeon!" How is this possible? Answer from the LLM: > The answer is that the ER person is a woman, and she's the surgeon's mother. Therefore, the "son" in the question refers to the surgeon…

I had to try this gem, it's my new benchmark! o4-mini-high also fails spectacularly, even after repeated feedback. However, 4.5 (the impractibly large demo model) gets it right:

It’s a nonsense twist on the classic lateral thinking puzzle:

The original puzzle goes: “A father and son are in a car accident. The father dies, and the son is rushed to the ER. The surgeon says, ‘I can’t operate on him—he’s my son.’” The intended answer to that puzzle challenges the listener to recognize the surgeon is the child’s mother, confronting implicit gender biases.

Your version humorously mangles it by swapping roles (“son and his man”) and objectifying the victims as cars, creating a logical absurdity. The sentence “I can’t operate on this car, he’s my surgeon!” doesn’t parse logically, indicating it’s a playful distortion rather than a coherent scenario

Post reply on HN