Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

21–30 of 211 posts

Re: Is the reversal curse in LLMs real?

#21

Post by thisgoesnowhere »

akoboldfrying wrote: post #9

Only skimmed this and didn't read the underlying paper, but I was surprised to see no mention of the fact that "A is B" often does not at all imply "B is A" in everyday language: A bird is an animal, but it's wrong to conclude that an arbitrary animal must be a bird.

The word an in this sentence (indefinite article) means that the word animal is non specific. So this structure does not follow from the original. "A is B" is not the same as "A is a B"

However, you are right that it's not always correct since "Poetry is Literature" and "Steam/Ice is H20" are both examples where this breaks down.

My annoying pedantry for the day.

Re: Is the reversal curse in LLMs real?

#22

Post by majormajor »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me.

It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on context or internalized-training of "Tom Cruise is the star of Mission Impossible" while not answering "Tom Cruise" to a much more general question like "who is covered in water" even if it had internalized a bit from a review like "there's a scene in Mission Impossible where Tom Cruise is covered in water..." unless it had context pointing it at that specific case.

Re: Is the reversal curse in LLMs real?

#23

Post by p1necone »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I don't think it's as simple as that - "the apple is red" is "single thing belongs to category", whereas "Olaf Scholz was the ninth Chancellor of Germany" is "single thing is single thing" - the latter is reversible, the former is not. I would expect a good language model to be able to parse both sentences correctly.

Re: Is the reversal curse in LLMs real?

#24

Post by rdedev »

tempestn wrote: post #7

> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them,…

Reading that article I can't help but see similarities between what we have in the human brain. We can easily form and recall short term memories, even using them to logically reason out facts, but loose such memories if it's not frequent enough. So it seems like the context prompt is similar to short term memory and what's missing now is a good way to transfer the short term to long term. There is a huge amount of assumptions going on here so take it with a huge amount of salt.

On another note I should have read the actual paper mentioned in the post critically instead of just skimming it. I completely missed the footnote about the reversal curse not being a problem if everything is present in the initial prompt

Re: Is the reversal curse in LLMs real?

#25

Post by LoganDark »

famouswaffles wrote: post #17
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look…

A prior (now-deleted) comment of yours put this in such a great way, and I'm sad that it got deleted:

> You could potentially describe how a game is played to GPT-4 without examples and get it playing it correctly but passing that same description into the training process of the model just gets you a model that can describe your game correctly.

Re: Is the reversal curse in LLMs real?

#26

Post by jetrink »

pmarreck wrote: post #13

I couldn't duplicate it in GPT4 here. It answered correctly, unless I posed it wrong: https://chat.openai.com/share/75d46a03-a223-4f3a-987d-f8fec3...

The paper is not talking about information provided in the context, only training and fine-tuning.

> In fairness, it’s also worth pointing out here that they’re making the claim that the reversal curse only applies to training and fine-tuning and not in-context – i.e., putting all your information inside a prompt. They point out in a footnote that you can put A to B data in a prompt and GPT-4 will make B to A connections just fine. Unfortunately, this was lost on many of the people covering the pre-print.

Re: Is the reversal curse in LLMs real?

#27

Post by zarzavat »

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

Re: Is the reversal curse in LLMs real?

#28

Post by foobarqux »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

But that's the problem isn't it? In some cases they are equivalent and in some cases they are not and a next-word-predictor needs to have "explicit" training data (i.e. it is not doing "reasoning") whereas a human can infer.

This isn't surprising if you think about how the machine actually works rather than treating it like a sacred magic box.

The default assumption for why there is any "success" for in-context learning should be that it's just picking a nearby token that "fits" not a process of logical deduction.

edit: The obvious band-aid-fix is to feed reversed sentences into the training data without telling anyone, after which LLM boosters will proclaim that LLMs "learned" to reverse logical implications.

Re: Is the reversal curse in LLMs real?

#30

Post by [deleted] »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

[deleted]
Post reply on HN