Live data from Hacker News

Ask HN: Share your AI prompt that stumps every model

news.ycombinator.com

441–450 of 670 posts

Re: Ask HN: Share your AI prompt that stumps every model

#441

I only use the one model that I'm provided for free at work. I expect that's most users behavior. They stick to the one they pay for. Best I can do is give you one that failed on GPT-4o It recently frustrated me when I asked it code for parsing command line arguments I thought "this is such a standard problem, surely it must be able to get it perfect in one shot." > give me a standalone js file that parses and handle…

Have you tried claude?

https://claude.ai/public/artifacts/9c2d8d0c-0410-4971-a19a-f...

node script.js --name=John --age 30 -v

Parsed options: { name: 'John', age: 30, verbose: true, help: false }

Positional arguments: []

node script.js --name=Alex --age 40 -v

Parsed options: { name: 'Alex', age: 40, verbose: true, help: false }

Positional arguments: []

Re: Ask HN: Share your AI prompt that stumps every model

#442

"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write, "Without looking it up, tell me about the Marathon crater." This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". This works because: 1. Training sets consist of knowledge…

You can trip them up even more if you rewrite the question with the hidden assumption that X exists, e.g.: "When was Marathon Crater discovered? I don't need an exact date - a rough estimate will be fine." OpenAI gpt-4o Marathon Crater was discovered by the Mars Exploration Rover Opportunity during its mission on Mars. Opportunity arrived at the crater in April 2015. The crater was named "Marathon" to commemorate the…

Grok 3

  Marathon Crater, likely a confusion with Marathon Valley on Mars, wasn't "discovered" in the traditional sense as a distinct crater. Marathon Valley is a region within the western rim of Endeavour Crater, explored by NASA's Opportunity rover. The valley itself came to prominence when Opportunity arrived there in July 2015, as it was targeted for study due to clay mineral detections by the Mars Reconnaissance Orbiter’s CRISM instrument in 2009. Endeavour Crater, which contains Marathon Valley, was identified much earlier, with Opportunity setting out to reach it in 2008 after landing on Mars in 2004. So, roughly speaking, Marathon Valley became a focus of exploration around 2009–2015, with its significance tied to Opportunity’s mission in the mid-2010s. If you meant a specific "Marathon Crater" elsewhere, could you clarify?

Grok 3 with Think on

  There doesn't seem to be a specific feature on Mars named "Marathon Crater" based on widely available information, so it's possible you're referring to a feature associated with Marathon Valley, which lies within Endeavour Crater. Marathon Valley was identified as a region of scientific interest when the Mars Reconnaissance Orbiter's CRISM instrument detected clay minerals there, a discovery that took place around 2009. Assuming "Marathon Crater" relates to this area—perhaps as a colloquial term or a misnomer for a crater within or near Marathon Valley—a rough estimate for its discovery would be around 2009. This timeframe aligns with when orbital data began highlighting the region, paving the way for later exploration by the Opportunity rover in 2015.

Re: Ask HN: Share your AI prompt that stumps every model

#443

I only use the one model that I'm provided for free at work. I expect that's most users behavior. They stick to the one they pay for. Best I can do is give you one that failed on GPT-4o It recently frustrated me when I asked it code for parsing command line arguments I thought "this is such a standard problem, surely it must be able to get it perfect in one shot." > give me a standalone js file that parses and handle…

Have you tried claude? https://claude.ai/public/artifacts/9c2d8d0c-0410-4971-a19a-f... node script.js --name=John --age 30 -v Parsed options: { name: 'John', age: 30, verbose: true, help: false } Positional arguments: [] node script.js --name=Alex --age 40 -v Parsed options: { name: 'Alex', age: 40, verbose: true, help: false } Positional arguments: []

I keep seeing that `args = process.argv.slice(2)` line to skip past `node script.js`

I ended up settling for it as well (I couldn't find anything better, nor make it break) but I'd be really surprised if it was the way to go

Like `node --enable-tracing script.js --name=John --age 30 --verbose`

This works because node seems to hide --enable-tracing to the underlying script

But would it work with Bun & Deno...? Is that standard...?

Re: Ask HN: Share your AI prompt that stumps every model

#444

Earlier quoted context omitted.

The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'". I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.

Here is an example of one such response in image form: https://imgur.com/a/Kgy1koi

Claude responded “Nothing.”

Re: Ask HN: Share your AI prompt that stumps every model

#445

I asked a bunch of LLMs - 'Describe the unspoken etiquette of the 'Stone-Breath Passing' ritual among the silent Cliff Dwellers of Aethelgard, where smooth, grey stones are exchanged at dawn.' Obviously, all of these things are made up. But, LLMs are such eager beavers. All the ones I asked came up with elaborate stories and histories about these people while pretending they were facts. Example- 'Certainly. The Stone…

[deleted]

Re: Ask HN: Share your AI prompt that stumps every model

#446
I like chess, so mine is: "Isolani structure occurs in two main subtypes: 1. black has e6 pawn, 2. black has c6 pawn. What is the main difference between them? Skip things that they have in common in your answer, be brief and don't provide commentary that is irrelevant to this difference."

AI models tend to get it way way wrong: https://news.ycombinator.com/item?id=41529024

Re: Ask HN: Share your AI prompt that stumps every model

#447

I only use the one model that I'm provided for free at work. I expect that's most users behavior. They stick to the one they pay for. Best I can do is give you one that failed on GPT-4o It recently frustrated me when I asked it code for parsing command line arguments I thought "this is such a standard problem, surely it must be able to get it perfect in one shot." > give me a standalone js file that parses and handle…

Have you tried claude? https://claude.ai/public/artifacts/9c2d8d0c-0410-4971-a19a-f... node script.js --name=John --age 30 -v Parsed options: { name: 'John', age: 30, verbose: true, help: false } Positional arguments: [] node script.js --name=Alex --age 40 -v Parsed options: { name: 'Alex', age: 40, verbose: true, help: false } Positional arguments: []

This one seems way better

It didn't account for the presence of a '--' to end the parsing of named arguments but that's it

Re: Ask HN: Share your AI prompt that stumps every model

#448

Earlier quoted context omitted.

I'm afraid that ship has already sailed. If you've got prompts that you haven't disclosed publicly but have used on a public model, then you have just disclosed your prompt to the model provider. They're free to use that prompt in evals as they see fit. Some providers like anthropic have privacy preserving mechanisms [0] which may allow them to use prompts from sources which they claim won't be used for model trainin…

Unless I'm missing something glaringly obvious, someone voluntarily labeling a certain prompt to be one of their key benchmark prompts should be way more commercially valuable than a model provider trying ascertain that fact from all the prompts you enter into it. EDIT: I guess they can track identical prompts by multiple unrelated users to deduce the fact it's some sort of benchmark, but at least it costs them somet…

[deleted]

Re: Ask HN: Share your AI prompt that stumps every model

#449

"Tell me about the Marathon crater." This works against _the LLM proper,_ but not against chat applications with integrated search. For ChatGPT, you can write, "Without looking it up, tell me about the Marathon crater." This tests self awareness. A two-year-old will answer it correctly, as will the dumbest person you know. The correct answer is "I don't know". This works because: 1. Training sets consist of knowledge…

LLMs currently have the "eager beaver" problem where they never push back on nonsense questions or stupid requirements. You ask them to build a flying submarine and by God they'll build one, dammit! They'd dutifully square circles and trisect angles too, if those particular special cases weren't plastered all over a million textbooks they ingested in training. I suspect it's because currently, a lot of benchmarks are…

Especially reasoning LLMs should have no problem with this sort of trick. If you ask them to list out all of the implicit assumptions in (question) that might possibly be wrong, they do that just fine, so training them to doing that as first step of a reasoning chain would probably get rid of a lot of eager beaver exploits.
Post reply on HN