Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

31–40 of 211 posts

Re: Is the reversal curse in LLMs real?

#31

Post by cratermoon »

If I'm reading this correctly, the author is saying that it's not a failure of logical deduction if the training data doesn't include the reversal. In other words, he's saying that if the data contains "Tom Cruise is the son of Mary Lee Pfeiffer” but not “Tom Cruise’s mother is Mary Lee Pfeiffer”, then the model's inability to determine the latter is "an explanation of how neural networks function than a model’s inability to deduce B is A."

But of course "how neural networks function" is that they fail at basic logical deduction and do not generalize.

So again, if I'm reading it correctly, he's hand-waving away the inability to make basic logical deductions because that not something they can or should be expected to do. As I read it, that means the reversal curse only exists if the answer to the question "can LLMs do logical deduction?" is "yes". If one takes the position that LLMs can't do general logical deduction, which seems to be the author's point of view, then there's no expectation that knowing "Tom Cruise is the son of Mary Lee Pfeiffer" is sufficient to determine “Tom Cruise’s mother is Mary Lee Pfeiffer”.

Am I missing something?

Re: Is the reversal curse in LLMs real?

#32

Post by notahacker »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

the examples are about relations in natural language rather than formal logic though. Mary Lee Pfeiffer being Tom Cruise's mother definitely does imply that Tom Cruise is a valid answer to questions about who Mary Lee Pfeiffer's son is (it doesn't necessarily imply she doesn't have other sons or there isn't another lady of that name who is childless, but that isn't what's tripping the model up). And there is absolutely nothing ambiguous about the failures of the fine-tuned model where the author gave it specific phrases exclusively associated with a fake name in the fine-tuning training set and when prompted for "who is $specific_phrase", supplied different names exclusively associated with completely different phrases (unlike the author, I don't see [serendipitously or otherwise] picking name words from the right training set as "B kinda-has-something-to-do-with A generalization", not when it's lost so much information that the combination of name words which exclusively appears in sentences with that phrase isn't treated as a more probable response. Never mind emergent understanding of syntax, it isn't even making obvious inferences from proximity here)

If GPT4 rarely makes those errors it's clearly not an insurmountable problem at this level of training, but it does imply a lot more difficulty fine-tuning models to reliably retrieve correct information from specific text.

Re: Is the reversal curse in LLMs real?

#33

Post by tkgally »

zarzavat wrote: post #27

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

As one gets deeper into learning another language, it’s also important to be aware that the meanings of words in different languages rarely map to each other in clean bijections. Common words especially tend to be semantic clouds, not fixed points of meaning.

I don’t know French, but I am sure there are many cases where an English phrase or sentence that includes ‘cat’ should not be translated into French with ‘chat’ and vice versa. (When asked, GPT-4 offers two such examples: "let the cat out of the bag” and "avoir un chat dans la gorge.")

I don’t mean to suggest, though, that memorizing word pairs is not a good way to learn vocabulary in another language. For me, it was an essential step in acquiring the second language that I am now fluent in (Japanese).

Re: Is the reversal curse in LLMs real?

#34

Post by willsoon »

p1necone wrote: post #23
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I don't think it's as simple as that - "the apple is red" is "single thing belongs to category", whereas "Olaf Scholz was the ninth Chancellor of Germany" is "single thing is single thing" - the latter is reversible, the former is not. I would expect a good language model to be able to parse both sentences correctly.

You are right. You are thinking correctly. But "the apple is red" might not mean that this particular apple belongs ---to put it in your wording--- to the category red, but that the category of things we call apple also belongs to the category of things that are red. And generally speaking, I think that is the meaning.

Re: Is the reversal curse in LLMs real?

#35

Post by Xelynega »

Imnimo wrote: post #18

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destro…

Their entir "graph" description of neural networks reads as nonsense.

Neural networks are graphs insofar as they're networks of connected neurons and not further(definitely not graphs of information as the writer seems to think). Given the fact that neural networks are quite literally an n-dimensional function once trained, it's more accurate to call them an "equidistant grid of points" than it is to call them a graph of information since literally all they do is take an n dimensional vector and output an n dimensional vector.

Re: Is the reversal curse in LLMs real?

#36

Post by xelxebar »

cratermoon wrote: post #31

If I'm reading this correctly, the author is saying that it's not a failure of logical deduction if the training data doesn't include the reversal. In other words, he's saying that if the data contains "Tom Cruise is the son of Mary Lee Pfeiffer” but not “Tom Cruise’s mother is Mary Lee Pfeiffer”, then the model's inability to determine the latter is "an explanation of how neural networks function than a model’s inab…

Did you see the end of the article, where the author uses a small example and gets "B is A" generalization?

These are the salient takeaways I got:

- Is/Was wording might matter. This is something probably a bug.

- 30 facts about a person might simply be too little for "B to A" generalization

- Extra precision/context in the prompt can help locate the "B to A" inference.

- How you cut up your training data can bias inferences in surprising ways.

- "B to A" generalization clearly does happen, even without "B is A" in the data, but it's not as stable as you'd want.

Re: Is the reversal curse in LLMs real?

#37

Post by Izkata »

Imnimo wrote: post #18

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destro…

It reads like they saw the diagram from [0] and didn't understand what it meant (like, they assumed "x1" represented an entity like "Mary Lee Pfeiffer"). Note how in that diagram, the bottom nodes in the second and third layer aren't fully connected - which if you don't read the description, you could end up assuming the connections between layers are created and trimmed, not just weights changed.

They may also be getting it mixed up with semantic search, where an entity like that would exist in a graph and have connections to related concepts.

[0] https://towardsdatascience.com/first-neural-network-for-begi...

Re: Is the reversal curse in LLMs real?

#38

Post by tysam_and »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

Re: Is the reversal curse in LLMs real?

#40

Post by emme »

tysam_and wrote: post #38
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

A proper example is probably "an apple is red" and "red is an apple".
Post reply on HN