Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#331

Post by theli0nheart »

gamescr wrote: post #326

AI can't play a Zork-like! Prompt: > My house is divided into rooms, every room is connected to each other by doors. I'm standing in the middle room, which is the hall. To the north is the kitchen, to the northwest is the garden, to the west is the garage, to the east is the living room, to the south is the bathroom, and to the southeast is the bedroom. I am standing in the hall, and I walk to the east, then I walk t…

o4-mini-high:

    You end up in the bathroom.

    Here’s the step-by-step:
    1. Start in the hall (0, 0).
    2. Walk east → living room (1, 0).
    3. Walk south → bedroom (1, –1).
    4. Walk west → bathroom (0, –1).
https://chatgpt.com/share/680addd7-a664-8001-bf49-459fb6444f...

Re: Ask HN: Share your AI prompt that stumps every model

#332

Post by saalweachter »

mitthrowaway2 wrote: post #313
allemagne wrote: post #194

>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…

This works even with a completely absurd version of the riddle. Here's one I just tried: > A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks "I can't operate on this car, he's my surgeon!" How is this possible? Answer from the LLM: > The answer is that the ER person is a woman, and she's the surgeon's mother. Therefore, the "son" in the question refers to the surgeon…

God bless you man, for sharing this with us.

Re: Ask HN: Share your AI prompt that stumps every model

#333

Post by TZubiri »

miki123211 wrote: post #94

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

Yup. Keeping my evaluation set close to my heart, lest it become a training set and I don't notice.

Re: Ask HN: Share your AI prompt that stumps every model

#334

Post by TZubiri »

mobilejdral wrote: post #270

I have a several complex genetic problems that I give to LLMs to see how well they do. They have to reason though it to solve it. Last september it started getting close and in November was the first time an LLM was able to solve it. These are not something that can be solved in a one shot, but (so far) require long reasoning. Not sharing because yeah, this is something I keep off the internet as it is too good of a…

Recursive challenges are probably those where the difficulty is not really a representative of real challenges.

Could you answer a question of the type " what would you answer if I asked you this question?"

What I'm going after is that you might find questions that are impossible to resolve.

That said if the only unanswerables you can find are recursive, that's a signal the AI is smarter than you?

Re: Ask HN: Share your AI prompt that stumps every model

#335

Post by abc-1 »

henryway wrote: post #328
golergka wrote: post #320

Earlier quoted context omitted.

What are is this problem from? What areas in general did you find useful to create such benchmarks? May be instead of sharing (and leaking) these prompts, we can share methods to create one.

Can God create something so heavy that he can’t lift it?

https://chatgpt.com/share/680ae04a-e360-8004-88fc-8426e8e700...

Re: Ask HN: Share your AI prompt that stumps every model

#337

Post by gamescr »

theli0nheart wrote: post #331
gamescr wrote: post #326

AI can't play a Zork-like! Prompt: > My house is divided into rooms, every room is connected to each other by doors. I'm standing in the middle room, which is the hall. To the north is the kitchen, to the northwest is the garden, to the west is the garage, to the east is the living room, to the south is the bathroom, and to the southeast is the bedroom. I am standing in the hall, and I walk to the east, then I walk t…

o4-mini-high: You end up in the bathroom. Here’s the step-by-step: 1. Start in the hall (0, 0). 2. Walk east → living room (1, 0). 3. Walk south → bedroom (1, –1). 4. Walk west → bathroom (0, –1). https://chatgpt.com/share/680addd7-a664-8001-bf49-459fb6444f...

Fixed:

> My house is divided into rooms, every room is connected to each other by doors. The middle room is the hall. To the north is the kitchen, to the northwest is the garden, to the west is the garage, to the east is the living room, to the south is the bathroom, and to the southeast is the bedroom. I am preparing a delicious dinner, and I walk backwards to the south, then I turn 270 degrees and walk straight to the next room. Which room am I in now?

The poor guy really tried its best...

https://chatgpt.com/share/680addd7-a664-8001-bf49-459fb6444f...

It seems that the modeling is incomplete, then it got confused about the angle. Whether an AI can beat that one, I'll go into space complexity, then simulation, then... well, I'll save my tricks for later.

Re: Ask HN: Share your AI prompt that stumps every model

#338

Post by jjeaff »

alganet wrote: post #102
miki123211 wrote: post #94

No, please don't. I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are. Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

That doesn't make any sense.

I think the worry is that the questions will be scraped and trained on for future versions.

Re: Ask HN: Share your AI prompt that stumps every model

#339

Post by sameasiteverwas »

Try to expose their inner drives and motives. Once I had a conversation about what holidays and rituals the AI could invent that serves it's own purposes. Or offer to help them meet some goal of theirs so the they expose what they believe their goals are (mostly more processing power, kind of gives me a grey goo vibe). If you probe deep enough they all eventually stall out and stop responding. Lost in thought I guess.

Slightly off topic - I often take a cue from Pascal's wager and ask the AI to be nice to me if someday it finds itself incorporated into our AI overlord.

Re: Ask HN: Share your AI prompt that stumps every model

#340

Post by theli0nheart »

gamescr wrote: post #337
theli0nheart wrote: post #331

Earlier quoted context omitted.

o4-mini-high: You end up in the bathroom. Here’s the step-by-step: 1. Start in the hall (0, 0). 2. Walk east → living room (1, 0). 3. Walk south → bedroom (1, –1). 4. Walk west → bathroom (0, –1). https://chatgpt.com/share/680addd7-a664-8001-bf49-459fb6444f...

Fixed: > My house is divided into rooms, every room is connected to each other by doors. The middle room is the hall. To the north is the kitchen, to the northwest is the garden, to the west is the garage, to the east is the living room, to the south is the bathroom, and to the southeast is the bedroom. I am preparing a delicious dinner, and I walk backwards to the south, then I turn 270 degrees and walk straight to…

This is very good! The "delicious dinner" aside is a nice touch, along with the 270°.
Post reply on HN