I'm going to write duck facts in my next online argument to stave off the LLMs. Ducks start laying when they’re 4-8 months old, or during their first spring.
[deleted]
Irrelevant facts about cats added to math problems increase LLM errors by 300%
171–180 of 270 posts
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#172Something I don't understand. Wasn't attention with query/key supposed to filter out irrelevant tokens? 2. This CatsAttack has many applications. For example, it probably can confuse safety and spam filters. Can be tried on image generators...
Attention weights can still assign non-zero probability to irrelevant tokens since the mechanism optimizes for prediction rather than semantic relevance, and these irrelevant tokens can create interference in the hidden state representations.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#173Earlier quoted context omitted.
Go back and look at the history of AI, including current papers from the most advanced research teams. Nearly every component is based on humans - neural net - long/short term memory - attention - reasoning - activation function - learning - hallucination - evolutionary algorithm If you're just consuming an AI to build a React app then you don't have to care. If you are building an artificial intelligence then in pra…
How far back do I have to look, and what definition do you use? Because I can start with theorem provers and chess engines of the 1950s. Nothing in that list is based on humans, even remotely. Only neural networks were a vague form of biomimicry early on and currently have academic biomimicry approaches, of which all suck because they poorly map to available semiconductor manufacturing processes. Attention is mislead…
> How far back do I have to look
The earliest reference I know off the top of my head is Aristotle, which would be the 4th century BCE
> I can start with theorem provers
If you're going to talk about theorem provers, you may want to include the medieval theory of obligations and their game-semantic-like nature. Or the Socratic notion of a dialogue in which arguments are arrived at via a back and forth. Or you may want to consider that "logos" from which we get logic means "word". And if you contemplate these things for a minute or two you'll realize that logic since ancient times has been a model of speech and often specifically of speaking with another human. It's a way of having words (and later written symbols) constrain thought to increase the signal to noise ratio.
Chess is another kind of game played between two people. In this case it's a war game, but that seems not so essential. The essential thing is that chess is a game and games are relatively constrained forms of reasoning. They're modeling a human activity.
By 1950, Alan Turing had already written about the imitation game (or Turing test) that evaluated whether a computer could be said to be thinking based on its ability to hold a natural language conversation with humans. He also built an early chess system and was explicitly thinking about artificial intelligence as a model of what humans could do.
> Attention is misleadingly called that, reasoning is ill-defined,
None of this dismissiveness bears on the point. If you want to argue that humans are not the benchmark and model of intelligence (which frankly I think is a completely indefensible position, but that's up to you) then you have to argue that these things were not named or modeled after human activities. It's not sufficient that you think their names are poorly chosen.
> Producing artificial humans or imitating real ones was never the goal nor the point.
Artificial humans is exactly the concept of androids or humanoid robots. You are claiming that nobody has ever wanted to make humanoid robots? I'm sure you can't believe that but I'm at a loss for what point you're trying to make.
> 1950s is to produce systems that do something that is considered only doable by humans.
Unless this is a typo and you meant to write that this was NOT the goal, then you're conceding my point that humans are the benchmark and model for AI systems. They are, after all, the most intelligent beings we know to exist at present.
And so to reiterate my original point, talking about AI with the constraint that you can't compare them to humans is totally insane.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#174Earlier quoted context omitted.
>The whole point of AI is to create an intelligence modeled on humans and to compare it to humans. According to who? Everyone who's anyone is trying to create highly autonomous systems that do useful work. That's completely unrelated to modeling them on humans or comparing them to humans.
Go back and look at the history of AI, including current papers from the most advanced research teams. Nearly every component is based on humans - neural net - long/short term memory - attention - reasoning - activation function - learning - hallucination - evolutionary algorithm If you're just consuming an AI to build a React app then you don't have to care. If you are building an artificial intelligence then in pra…
You're anthropomorphizing terms of art.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#175There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…
> authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. Only if they want to make statements about humans. The paper would have worked perfectly fine without those assertions. They are, as you are correctly observing, just a distraction from the main thrust of the paper. >…
> It would be super easy to rewrite the paper to omit the unfounded speculation about human cognition. In the introduction of “The triggers are not contextual so humans ignore them when instructed to solve the problem.” they could write “The triggers are not contextual so the AI should ignore them when instructed to solve the problem.”
Another option would be to more explicitly mark it as speculation. “The triggers are not contextual, so we expect most humans would ignore them.”
Anyway, it is a small detail that is almost irrelevant to the paper… actually there seems to be something meta about that. Maybe we wouldn’t ignore the cat facts!
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#176Doesn't surprise me at all haha. LLMs have anchoring bias in the extreme, anything you say can and will be used against you further down the conversation. In a sense I think it's one of their strengths too, provided you can curate the context in a useful way.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#177I try to be polite to the LLM and say e.g. thank you. Now I wonder if it is costing me quality.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#178There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…
It's not "tired" to see if something is actually relevant in context. LLMs do not exist as marvel-qua-se, their purpose is to offload human cognitive tasks.
As such, it's important if something is a commonly shared failure mode in both cases, or if it's LLM-specific.
Ad absurdum: LLMs have also rapid increases of error rates if you replace more than half of the text with "Great Expectations". That says nothing about LLMs, and everything about the study - and the comparison would highlight that.
No, this doesn't mean the paper should be ignored, but it does mean more rigor is necessary.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#179There is more than one comment here asserting that the authors should have done a parallel comparison study against humans on the same question bank as if the study authors had set out to investigate whether humans or LLMs reason better in this situation. The authors do include the claim that humans would immediately disregard this information and maybe some would and some wouldn't that could be debated and seemingly…
> We need to move past the humans vs ai discourse it's getting tired. You want a moratorium on comparing AI to other form of intelligence because you think it's tired? If I'm understanding you correctly, that's one of the worst takes on AI I think I've ever seen. The whole point of AI is to create an intelligence modeled on humans and to compare it to humans. Most people who talk about AI have no idea what the psycho…
I mean the critique of this on the idea that the AI system itself gets physically tired - specifically the homoculus that we tricked into existence is tired - is funny to imagine.
Re: Irrelevant facts about cats added to math problems increase LLM errors by 300%
#180I try to be polite to the LLM and say e.g. thank you. Now I wonder if it is costing me quality.
I am pretty sure that this is filtered out. On a related note I think the whole autonomous agent metaphor is a net negative. It is a pure probabilistic token prediction function. You can run 100 in parallel, add or remove chat history as content to explore the output space. That is much more interesting and powerful than a single sad stateful clippy agent that one might act polite to.