Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#91

Post by paulcole »

robviren wrote: post #68

"If I can dry two towels in two hours, how long will it take me to dry four towels?" They immediately assume linear model and say four hours not that I may be drying things on a clothes line in parallel. It should ask for more context and they usually don't.

How long has it been since you’ve tried this?

Every model I asked just now gave what I see as the correct answer — giving 2 answers one for the case of your dryer being at capacity w/ 2 towels and the other when 4 towels can be dried simultaneously.

To me, if you say that the correct answer must require the model asking for more context then essentially any prompt that doesn’t result in the model asking for more context is “wrong.”

Re: Ask HN: Share your AI prompt that stumps every model

#92

Post by ryankrage77 »

moffkalast wrote: post #64
falcor84 wrote: post #2

You might want to get the ball rolling by sharing what you already have

Also, ones that can't be solved at a glance by humans don't count. Like this horrid ambiguous example from SimpleBench I saw a while back that's just designed to confuse: John is 24 and a kind, thoughtful and apologetic person. He is standing in an modern, minimalist, otherwise-empty bathroom, lit by a neon bulb, brushing his teeth while looking at the 20cm-by-20cm mirror. John notices the 10cm-diameter neon lightbul…

I'd argue that's a pretty good test for an LLM - can it overcome the red herrings and get at the actual problem?

Re: Ask HN: Share your AI prompt that stumps every model

#93

Post by reginald78 »

ks2048 wrote: post #62

I don't know if it stumps every model, but I saw some funny tweets asking ChatGPT something like "Is Al Pacino in Heat?" (asking if some actor or actress in the film "Heat") - and it confirms it knows this actor, but says that "in heat" refers to something about the female reproductive cycle - so, no, they are not in heat.

I believe it was GoogleAI in search but it was worse than that. Some asked it if Angelina Jolie was in heat. The tone started kind of insulting like the user was a sexist idiot for thinking human women go into heat like animals, then went back and forth saying she is still fertile at her age and also that her ovaries had been removed. It was funny because it managed to be arrogant, insulting, kind of creepy and gross and logically inconsistent while not even answering the question.

Angelina Jolie was not in Heat (1995). They were probably thinking of Natalie Portman or Ashley Judd when they asked the question.

Re: Ask HN: Share your AI prompt that stumps every model

#94

Post by miki123211 »

No, please don't.

I think it's good to keep a few personal prompts in reserve, to use as benchmarks for how good new models are.

Mainstream benchmarks have too high a risk of leaking into training corpora or of being gamed. Your own benchmarks will forever stay your own.

Re: Ask HN: Share your AI prompt that stumps every model

#96

Post by fragmede »

I want to know as well! Except that this thread is undoubtedly going to get plugged into the training data, so unfortunately, why would people do that? For mine that worked before the ChatGPT 4.5, it was the river crossing problem. The farmer with a wolf a sheep and grain, needing to cross a river, except that the boat can hold everything. Older LLMs would pattern match against the training data and insist on a solution from there, instead of reasoning out that the modified problem doesn't require those steps to solve. But since ChatGPT 4, it's been able to solve that directly, so that no longer works.

Re: Ask HN: Share your AI prompt that stumps every model

#98

Post by mNovak »

moffkalast wrote: post #64
falcor84 wrote: post #2

You might want to get the ball rolling by sharing what you already have

Also, ones that can't be solved at a glance by humans don't count. Like this horrid ambiguous example from SimpleBench I saw a while back that's just designed to confuse: John is 24 and a kind, thoughtful and apologetic person. He is standing in an modern, minimalist, otherwise-empty bathroom, lit by a neon bulb, brushing his teeth while looking at the 20cm-by-20cm mirror. John notices the 10cm-diameter neon lightbul…

At a glance, it sounds like John is the bald man? If we're treating this as a riddle, it doesn't seem incomprehensible. Whether riddles are a fair test is another question.

Re: Ask HN: Share your AI prompt that stumps every model

#100

Post by spuz »

moffkalast wrote: post #64
falcor84 wrote: post #2

You might want to get the ball rolling by sharing what you already have

Also, ones that can't be solved at a glance by humans don't count. Like this horrid ambiguous example from SimpleBench I saw a while back that's just designed to confuse: John is 24 and a kind, thoughtful and apologetic person. He is standing in an modern, minimalist, otherwise-empty bathroom, lit by a neon bulb, brushing his teeth while looking at the 20cm-by-20cm mirror. John notices the 10cm-diameter neon lightbul…

Yeah I really dislike this kind of question from SimpleBench. I've suggested many improvements to some of the publicly available questions but not had a good response.

I think the correct answer to the question above should be something like, "are you sure the question is correct because it's not clear whether John and the bald man are the same person" but of course an LLM would be marked down if it was inclined to respond in this way.

Post reply on HN