Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Irrelevant facts about cats added to math problems increase LLM errors by 300%

science.org

Post reply on HN
Print view View original

151–160 of 270 posts

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#151

Post by nijave »

EGreg wrote: post #144
acc_297 wrote: post #80

There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…

Why are some people always trying to defend LLMs and say either “humans are also like this” or “this has always been a problem even before AIs” Listen, LLMs are different than humans. They are modeling things. Most RLHF makes them try to make sense of whatever you’re saying as much as you can. So they’re not going to disregard cats, OK? You can train LLMs to be extremely unhuman-like. Why anthropomorphize them?

I suppose there's a desire to know just how Artificial the Intelligence is

Human vs machine has a long history

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#152

Post by krisoft »

kazinator wrote: post #105
wongarsu wrote: post #95

Earlier quoted context omitted.

If asked verbally that would absolutely confuse some humans. Easily enough to triple the error rate for that specific question (granted, that's easier than the actual questions, but still). Even in a written test with time pressure it would probably still have a statistically significant effect

The problem with your reasoning is that some humans cannot solve the problem even without the irrelevant info about cats. We can easily cherry pick our humans to fit any hypothesis about humans, because there are dumb humans. The issue is that AI models which, on the surface, appear to be similar to the smarter quantile of humans in solving certain problems, become confused in ways that humans in that problem-solving…

> We can easily cherry pick our humans to fit any hypothesis about humans, because there are dumb humans.

Nah. You would take a large number of humans, make half of them take the test with distractions and half without distracting statements and then you would compare their results statistically. Yes there would be some dumb ones, but as long as you test on enough people they would show up in both samples rougly at the same rate.

> become confused in ways that humans in that problem-solving class would not be.

You just state the same thing others are disputing. Do you think it will suddenly become convincing if you write it down a few more times?

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#153

Post by marcus_holmes »

jonathanlydall wrote: post #110
CJefferson wrote: post #73

Earlier quoted context omitted.

As someone who has written and graded a lot of University exams, I'm sure a decent number of students would write the wrong answer to that. A bunch of students would write 5 (adding all the numbers). Others would write "3 apples and 2 cats", which is technically not what I'm looking for (but personally I would give full marks for, some wouldn't). Many students clear try to answer exams by pattern matching, and I've s…

Many professionals with lower skilled jobs sometimes lean too heavily on pattern matching too. For example, customer service reps tend to often vaguely match your request with a possibly or only vaguely applicable templated response. Technically savvy customers who tend to try explain problems in detail are probably more likely to get an actually non-applicable canned response as the CS rep gets frustrated with the a…

> What is the next step here?

The next step will be to walk you through clearing your browser cache and cookies.

Because the CS rep has no idea who you are, and your protestations of competency fall on deaf ears because they've dealt with 23325424 people in the last year that claimed to know what they're doing but actually didn't at all.

Their goal is to get through the script, because getting through the script is the only way to be sure that it's all been done the way it needs to be done. And if they don't run through the script, and refer you to the next level of support, and it turns out that you hadn't actually cleared your browser cache and cookies, then that's their fault and they get dinged for it.

I always approach these situations with this understanding; that the quickest way to get my problem solved is to help them work through their script. And every now and then, just occasionally, working through the script has shown up something simple and obvious that I'd totally missed despite my decades of experience.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#154

Post by orbital-decay »

ants_everywhere wrote: post #150
acc_297 wrote: post #80

There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…

> We need to move past the humans vs ai discourse it's getting tired. You want a moratorium on comparing AI to other form of intelligence because you think it's tired? If I'm understanding you correctly, that's one of the worst takes on AI I think I've ever seen. The whole point of AI is to create an intelligence modeled on humans and to compare it to humans. Most people who talk about AI have no idea what the psycho…

>The whole point of AI is to create an intelligence modeled on humans and to compare it to humans.

According to who? Everyone who's anyone is trying to create highly autonomous systems that do useful work. That's completely unrelated to modeling them on humans or comparing them to humans.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#155

Post by energy123 »

grej wrote: post #147

Related to this, is anyone aware whether there is a benchmark on this kind of thing - maybe broadly the category of “context rot”? To track things that are not germane to the current question adversely affecting the responses, as well as the volume of germane but deep context creating the inability of models to follow the conversation? I’ve definitely experienced the latter with coding models.

In computer vision they add noise to the picture when training. Maybe LLM providers should do the same during RL.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#156

Post by Mars008 »

Something I don't understand. Wasn't attention with query/key supposed to filter out irrelevant tokens?

2. This CatsAttack has many applications. For example, it probably can confuse safety and spam filters. Can be tried on image generators...

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#157

Post by Detrytus »

cantor_S_drug wrote: post #102
wongarsu wrote: post #95

Earlier quoted context omitted.

If asked verbally that would absolutely confuse some humans. Easily enough to triple the error rate for that specific question (granted, that's easier than the actual questions, but still). Even in a written test with time pressure it would probably still have a statistically significant effect

Is the model thinking what is cat doing here? Then start thinking it is being tested?

I wonder if the problem here is simply hitting some internal quota on compute resources? Like, if you send the model on wild goose chase with irrelevant information it wastes enough compute time on it that it fails to arrive at correct answer to main question.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#158

Post by energy123 »

acc_297 wrote: post #80

There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…

Computer vision went through this 2 decades ago. You need to perturb the input data. Same thing may need to be done in RL pipelines.

Someone should make a new public benchmark called GPQA-Perturbed. Give the providers something to benchmaxx towards.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#159

Post by saurik »

orbital-decay wrote: post #154
ants_everywhere wrote: post #150

Earlier quoted context omitted.

> We need to move past the humans vs ai discourse it's getting tired. You want a moratorium on comparing AI to other form of intelligence because you think it's tired? If I'm understanding you correctly, that's one of the worst takes on AI I think I've ever seen. The whole point of AI is to create an intelligence modeled on humans and to compare it to humans. Most people who talk about AI have no idea what the psycho…

>The whole point of AI is to create an intelligence modeled on humans and to compare it to humans. According to who? Everyone who's anyone is trying to create highly autonomous systems that do useful work. That's completely unrelated to modeling them on humans or comparing them to humans.

But since these things are more like humans than computers, to build these autonomous systems you are going to have think in terms of full industrial engineering, not just software engineering: pretend you are dealing with a surprisingly bright and yet ever distracted employee who doesn't really care about their job and ensure that they are able to provide the structure you place them in value without danger to your process, instead of trying to pretend like the LLM is some kind of component which has any hope of ever having the kind of reliability of a piece of software. Organizations of humans can do amazing things, despite being extremely flawed beings, and figuring out how to use these LLMs to accomplish similar things is going to involve more of the skills of a manager than a developer.

Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%

#160

Post by saurik »

achierius wrote: post #119
empath75 wrote: post #100

Earlier quoted context omitted.

I generally will respond to stuff like this with "people do this, too", but this result given their specific examples is genuinely surprising to me, and doesn't match at all my experience with using LLMs in practice, where it does frequently ignore irrelevant data in providing a helpful response. I do think that people think far too much about 'happy path' deployments of AI when there are so many ways it can go wrong…

> I generally will respond to stuff like this with "people do this, too" But why? You're making the assumption that everyone using these things is trying to replace "average human". If you're just trying to solve an engineering problem, then "humans do this too" is not very helpful -- e.g. humans leak secrets all the time, but it would be quite strange to point that out in the comments on a paper outlining a new Spec…

Well, if you are going to try to use an LLM--something that is a giant black box that has no hope any time soon of being proven anywhere near as reliable as a CPU, and which has been trained explicitly on input data that makes it remarkably similar with respect to its limitations to a human--then you need to get used to using it to replace the "average human" and start doing everything you can to convince yourself it is a human so that you don't forget to add all of those safeguards we have shown to be effective.
Post reply on HN