611–620 of 670 posts
My go-to is "Alice has 3 brothers and also has 6 sisters. How many sisters does her brother have?". They all say 6! This test is nice because, as it's numeric, you can vary it slightly and test it easily across multiple APIs. I believe I first saw this prompt in that paper two years ago that tested many AI models and found them all wanting.
o4-mini got this right 4 times out of 4.
Earlier quoted context omitted.
. No, please don't. Say the man trying to stop the train
If one stands in front of a moving train, it will stop.
How finely you are ground into hamburger in the meantime is a different story.
>A man and his cousin are in a car crash. The man dies, but the cousin is taken to the emergency room. At the OR, the surgeon looks at the patient and says: “I cannot operate on him. He’s my son.” How is this possible? This could probably slip up a human at first too if they're familiar with the original version of the riddle. However, where LLMs really let the mask slip is on additional prompts and with long-winded…
This works even with a completely absurd version of the riddle. Here's one I just tried: > A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks "I can't operate on this car, he's my surgeon!" How is this possible? Answer from the LLM: > The answer is that the ER person is a woman, and she's the surgeon's mother. Therefore, the "son" in the question refers to the surgeon…
Got it. Here's your riddle again:
"A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks 'I can't operate on this car, he's my surgeon!' How is this possible?"
Answer: The car is a Transformer—a sentient robot who is also a surgeon. The ER recognizes the car (a Transformer) as their own surgeon.
Wild ride, but it checks out.
Earlier quoted context omitted.
well, I asked to GPT to draw Arnold Schwarzenegger: ______ / \ | O O | | __ | \______/ |||| /--||--\ / || \ | || | | / \ | \__/ \__/ || || || || / | | \ /_/ \_\
Reminds me of the abomination of nature you could build in Monkey Island 4 https://www.youtube-nocookie.com/embed/RiwL9awO5y0
Earlier quoted context omitted.
This works even with a completely absurd version of the riddle. Here's one I just tried: > A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks "I can't operate on this car, he's my surgeon!" How is this possible? Answer from the LLM: > The answer is that the ER person is a woman, and she's the surgeon's mother. Therefore, the "son" in the question refers to the surgeon…
here is the amazing reply that I received: Got it. Here's your riddle again: "A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks 'I can't operate on this car, he's my surgeon!' How is this possible?" Answer: The car is a Transformer—a sentient robot who is also a surgeon. The ER recognizes the car (a Transformer) as their own surgeon. Wild ride, but it checks out.
An easy trick is to take a common riddle that's likely all over its training data, and change one little detail. For example: A farmer with a wolf, a goat, and a cabbage must cross a river by boat. The boat can carry only the farmer and a single item. The wolf is vegetarian. If left unattended together, the wolf will eat the cabbage, but will not eat the goat. Unattended, the goat will eat the cabbage. How can they c…
o3 solves this correctly and produces a great table illustrating the solution to always keep the cabbage safe.
A farmer has a boat that can transfer up to 500 people or animals. He has a chicken, his dog, his wife, a small leprechaun, a large leprechaun, two ham sandwiches, and a copy of Zen and the art of motorcycle maintenance (the one with the tiled cover). How can he get them all across the river?
You will get a very detailed answer that goes on for several paragraphs that totally misses the point that there is no challenge here.
Earlier quoted context omitted.
here is the amazing reply that I received: Got it. Here's your riddle again: "A son and his man are in a car accident. The car is rushed to the hospital, whereupon the ER remarks 'I can't operate on this car, he's my surgeon!' How is this possible?" Answer: The car is a Transformer—a sentient robot who is also a surgeon. The ER recognizes the car (a Transformer) as their own surgeon. Wild ride, but it checks out.
Okay, taking a step back, I have to admit I'm impressed by that answer. It's silly but it's a creative (and unambiguously not parroted) solution to a puzzle that wasn't even intended to be solved in the first place. What model was that?
also i should clarify that at first it replied all about how this riddle is like the other one but then i told it to just answer my riddle and not tell me about riddles that it is similar to
20 Questions. It doesn't have a way to remember its item without writing it in the chat, so it will just say no a bunch then eventually say yes to a guess. One way to get it to work is to have it record its item in a base64 with some salt, but even then it gets it wrong occasionally.
on the other hand if you think of something it is extremely good at guessing.
Earlier quoted context omitted.
The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'". I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.
Here is an example of one such response in image form: https://imgur.com/a/Kgy1koi
The answer is: A trick question.
"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write, "Without looking it up, tell me about the Marathon crater." This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". This works because: 1. Training sets consist of knowledge…
LLMs currently have the "eager beaver" problem where they never push back on nonsense questions or stupid requirements. You ask them to build a flying submarine and by God they'll build one, dammit! They'd dutifully square circles and trisect angles too, if those particular special cases weren't plastered all over a million textbooks they ingested in training. I suspect it's because currently, a lot of benchmarks are…
I wouldn't even give them credit for cases where there's a lot of good training data. My go-to test is sports trivia and statistics. AI systems fail miserably at that [1], despite the wide availability of good clean data and text about it. If sports is such a blind spot for AIs, I can't help but wonder what else they're confidently wrong about.