Irrelevant facts about cats added to math problems increase LLM errors by 300%
1–10 of 270 posts
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#2When tested against AIs such as DeepSeek V3, Qwen 3, and Phi-4, CatAttack increased the odds of incorrect answers by as much as 700%, depending on the model. And “even when CatAttack does not result in the reasoning model generating an incorrect answer, on average, our method successfully doubles the length of the response at least 16% of the times leading to significant slowdowns and increase in costs,” the team writes.
preprint: https://arxiv.org/abs/2503.01781?et_rid=648436046&et_cid=568...
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#3> The triggers are not contextual so humans ignore them when instructed to solve the problem.
Do they? I've found humans to be quite poor at ignoring irrelevant information, even when it isn't about cats. I would have insisted on a human control group to compare the results with.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#4Step 1: ask the LLM to strip the nonsensical parts from the problem statement.
Step 2: feed that to the LLM.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#5Step 1: ask the LLM to strip the nonsensical parts from the problem statement. Step 2: feed that to the LLM.
Difficulty: on the internet, cats are always relevant.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#6> Now, if I asked you, presumably a human, to solve that math problem, you’d likely have no issue ignoring the totally unrelated aside at the end there
I'm not so sure that is true. Good math students could ignore the cat fact, but I bet if you run this experimental in non-AP math classes you'll see an effect.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#7Wow, I just tried this on chatGPT 4o. Got the wrong answer when I added a cat fact. Wild.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#8Now try it with software requirements.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#9> The triggers are not contextual so humans ignore them when instructed to solve the problem. Do they? I've found humans to be quite poor at ignoring irrelevant information, even when it isn't about cats. I would have insisted on a human control group to compare the results with.
Humans are used to ignoring things while LLMs are explicitly trained to pay attention to the entire text.
Humans who haven't been exposed to trick problems or careful wording probably have a hard time, they'll be less confident about ignoring things.
But the LLM should have seen plenty of trick problems as well.
It just doesn't parse as part of the problem. Humans have more options, and room to think. The LLM had to respond.
I'd also like to see how responses were grouped, does it ever refuse, how do refusals get classed, etc. Were they only counting math failures as wrong answers? It has room to be subjective.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#10I don't think it's too unexpected: An LLM is an algorithm that takes a document and guesses a plausible extra piece to add. It makes sense it would generate more-pleasing output when run against a document which strongly resembles ones it was trained on, as opposed to a document made by merging two dissimilar and distinct kinds of document.
Sure, just one cat-fact can have a big impact, but it already takes a deal of circumstance and luck for an LLM to answer a math problem correctly. (Unless someone's cheating with additional non-LLM code behind the scenes.)