Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#291

Post by School-Cotton »

ls612 wrote: post #209
rybosworld wrote: post #176

Earlier quoted context omitted.

Tuning the model output to perform better on certain prompts is not the same as improving the model. It's valid to worry that the model makers are gaming the benchmarks. If you think that's happening and you want to personally figure out which models are really the best, keeping some prompts to yourself is a great way to do that.

Who’s going out of their way to optimize for random HNers informal benchmarks?

They're probably not going out of their way, but I would assume all mainstream models have HN in their training set.

Re: Ask HN: Share your AI prompt that stumps every model

#292

Post by stordoff »

mohsen1 wrote: post #6

A ball costs 5 cents more than a bat. Price of a ball and a bat is $1.10. Sally has 20 dollars. She stole a few balls and bats. How many balls and how many bats she has? All LLMs I tried miss the point that she stole things and not bought them

GPT-4o claims "This implies she did not pay the full $20. The total cost of the balls and bats she has is less than $20, but she still has items worth up to $20.", then bruteforces an 'answer' of "Balls = 25 Bats = 13".

It also managed to get the prices of the ball/bat wrong, presumably because it's using the more typical riddle:

> Ball = x dollars

> Bat = x + $0.05 (since it’s 5 cents more than the ball)

https://chatgpt.com/share/680ac88c-22d4-8011-b642-0397a01ec3...

Re: Ask HN: Share your AI prompt that stumps every model

#293

Post by wolfgang42 »

CamperBob2 wrote: post #256
Certified wrote: post #56

Earlier quoted context omitted.

GPT 4.5 seems to get it right, but then repeat the 700 pounds "A woodchuck would chuck as much wood as a woodchuck could chuck if a woodchuck could chuck wood. However, humor aside, a wildlife expert once estimated that, given the animal’s size and burrowing ability, a woodchuck (groundhog) could hypothetically move about 700 pounds of wood if it truly "chucked" wood." https://chatgpt.com/share/680a75c6-cec8-8012-a57…

That answer is exactly right, and those who say the 700 pound thing is a hallucination are themselves wrong: https://chatgpt.com/share/680aa077-f500-800b-91b4-93dede7337...

Linking to ChatGPT as a “source” is unhelpful, since it could well have made that up too. However, with a bit of digging, I have confirmed that the information it copied from Wikipedia here is correct, though the AP and Spokane Times citations are both derivative sources; Mr. Thomas’s comments were first published in the Rochester Democrat and Chronicle, on July 11, 1988: https://democratandchronicle.newspapers.com/search/results/?...

Re: Ask HN: Share your AI prompt that stumps every model

#294

Post by lxe »

vunderba wrote: post #103
thatjoeoverthr wrote: post #21

"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write, "Without looking it up, tell me about the Marathon crater." This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". This works because: 1. Training sets consist of knowledge…

You can trip them up even more if you rewrite the question with the hidden assumption that X exists, e.g.: "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine." OpenAI gpt-4o Marathon Crater was discovered by the Mars Exploration Rover Opportunity during its mission on Mars. Opportunity arrived at the crater in April 2015. The crater was named "Marathon" to commemorate the…

Sonnet 3.7:

    > tell me about the marathon crater

    I'd be happy to tell you about Marathon Crater. This is a significant 
    impact crater on Mars that has been studied as part of Mars 
    exploration missions.

    Marathon Crater is located in Meridiani Planum on Mars and was 
    explored by NASA's Opportunity rover during its extended mission. 
    The crater is named after Marathon, a town in Texas, following 
    the naming convention of using Earth locations for features in 
    Meridiani Planum.... etc etc

Re: Ask HN: Share your AI prompt that stumps every model

#295

Post by torial »

mdp2021 wrote: post #182

Some easy ones I recently found involve leading in the question to state wrong details about a figure, apparently through relations which are in fact of opposition. So, you can make them call Napoleon a Russian (etc.) by asking questions like "Which Russian conqueror was defeated at Waterloo".

Some researchers were testing various Legal AI models and one of their questions was about why a Supreme Court justice who dissented in the case (the justice in this case assented).

Re: Ask HN: Share your AI prompt that stumps every model

#296

Post by vitaflo »

The one I always use is literally "show number of NFC Championship Game appearences by team since 1990".

The only AI that has ever gotten the answer right was Deepseek R1. All the rest fail miserably at this one. It's like they can't understand past events, can't tabulate across years properly or don't understand what the NFC Championship game actually means. Many results "look" right, but they are always wrong. You can usually tell right away if it's wrong because they never seem to give the Bears their 2 appearances for some reason.

Re: Ask HN: Share your AI prompt that stumps every model

#297

Post by JCattheATM »

jaffa2 wrote: post #273
Der_Einzige wrote: post #252

Earlier quoted context omitted.

[flagged]

I never heard of this phrase before ( i had heard the concept , i think this is similar to the paperclip problem) but now in 2 days ive heard it twice here and on youtube. Rokokos basilisk.

It's a completely nonsense argument and should be dismissed instantly.

Re: Ask HN: Share your AI prompt that stumps every model

#300

Post by creata »

buzzy_hacker wrote: post #152

"Aaron and Beren are playing a game on an infinite complete binary tree. At the beginning of the game, every edge of the tree is independently labeled A with probability p and B otherwise. Both players are able to inspect all of these labels. Then, starting with Aaron at the root of the tree, the players alternate turns moving a shared token down the tree (each turn the active player selects from the two descendants…

OpenAI's o4-mini got the right answer after "thinking" for 29 seconds. It's a straightforward puzzle, though: no creativity involved.
Post reply on HN