Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

101–110 of 211 posts

Re: Is the reversal curse in LLMs real?

#101

Post by omoide »

TeMPOraL wrote: post #95
zarzavat wrote: post #27

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

Not just languages. It's the case with everything . Easy to spot when you're tutoring someone. You can see they learned "force is mass times acceleration" or "for( ... ) is how you make the same code run multiple times" - they can tell you that when quizzed! But you know they haven't comprehended it until they can reverse it - "I need to compute the mass of the object, which I see accelerating this much under this fo…

Interesting; your message hints at a sharp gap between using language stochastically (eg being able to repeat what you've heard) vs using language to express deeper knowledge/understanding.

A common argument is that LLM by their nature only model the former, not the latter.

Re: Is the reversal curse in LLMs real?

#102

Post by karatinversion »

bloak wrote: post #82
School-Cotton wrote: post #63

Earlier quoted context omitted.

> but that the category of things we call apple also belongs to the category of things that are red No, in English this would be “apples are red”, not “the apple is red”.

I don't disagree though perhaps it's worth mentioning that sentences such as "the great spotted woodpecker is a medium-sized woodpecker" are used in English with the meaning "great spotted woodpeckers are medium-sized woodpeckers" so it seems to me that "apples are red" is grammatically a possible meaning of "the apple is red" even if it would be stylistically and pragmatically so weird that nobody would ever do that…

Indeed, and even with apples, we can say "the apple is native to Central Asia" without meaning a particular apple.

Re: Is the reversal curse in LLMs real?

#103

Post by TeMPOraL »

tysam_and wrote: post #38
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

True, but only in the implicit, very narrow world typically assumed when doing logic exercises. "Red is the apple", obviously, because our whole world consists of "The apple is red" and some inference rules. In that world, "Red is the" can only be completed by "apple". Drop the narrowing though, and suddenly there's a lot of other completions, many of them much better ones (I'd guess "blood" will be one of the more frequent ones - "red is the blood of" shows up a lot in songs and poems).

Or in short: LLMs will handle "The apple is red. Red is the " just fine, because the first part is heavily biasing the answer. They won't often complete "Red is the " alone with "apple", because why should they?

Re: Is the reversal curse in LLMs real?

#104

Post by TeMPOraL »

omoide wrote: post #101
TeMPOraL wrote: post #95

Earlier quoted context omitted.

Not just languages. It's the case with everything . Easy to spot when you're tutoring someone. You can see they learned "force is mass times acceleration" or "for( ... ) is how you make the same code run multiple times" - they can tell you that when quizzed! But you know they haven't comprehended it until they can reverse it - "I need to compute the mass of the object, which I see accelerating this much under this fo…

Interesting; your message hints at a sharp gap between using language stochastically (eg being able to repeat what you've heard) vs using language to express deeper knowledge/understanding. A common argument is that LLM by their nature only model the former, not the latter.

My point is that LLMs can do both, the same way humans can. We don't learn or recall reversals automatically either - we memorize them separately, or run an inference step (which, done often enough, leads to memorization). Reversals aren't free - not for LLMs, and not for our own minds.

Re: Is the reversal curse in LLMs real?

#106

Post by thfuran »

tysam_and wrote: post #50
majormajor wrote: post #22

Earlier quoted context omitted.

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on…

It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help. Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bi…

>it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.

But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?

Re: Is the reversal curse in LLMs real?

#107

Post by TeMPOraL »

user_named wrote: post #89
TeMPOraL wrote: post #85

Earlier quoted context omitted.

IDK, I still some of it boil down to "B -> A" being extremely ambiguous. Other than the annoying Tom Cruise example (I finally know where it came from!), I've seen people frequently bring up "The color of the sky is blue" vs. "Blue is the color of the sky" - but this illustrates my hypothesis perfectly. In training data (and in the totality of what humans ever said or wrote), "Color of the sky is " is almost certain…

Nah. The context of all of this is people claiming LLMs are AI. If so they would be able to reverse A is B, and also know when not to, so your example is irrelevant. The paper shows that it cannot reverse. Then the blog posts goes around different disingenuous ways of making excuses for why it can't reverse at the same time as trying to show that it can reverse in the Tom Cruise training example (extremely disingenuo…

I still call bullshit on this. Humans don't do reversals for free either. We may memorize the reversal, or first, infer it. That's an extra step, often very cognitively challenging. I'd like to see experiments showing how "reversal curse" fares when LLM is allowed to make the inference. Like:

"Please complete the sentence: [your B->A] - but reason through the process first. Start with identifying the subject, then write a high-level summary of what you know about the subject, and only then attempt to complete the original sentence."

I'd expect something like this to suddenly score much better. And I don't consider this cheating - because I think the "AI-ness" quality of LLMs shouldn't be measured against the workings of a human mind, but rather the workings of the inner voice in the human's mind.

Re: Is the reversal curse in LLMs real?

#109

Post by weinzierl »

"As a side note: I want to point out that I’m not aware of any examples of capabilities that can be done with prompting a model like GPT-4 that it can’t be trained for. This is why I’m a little skeptical."

While it is just a side-note in the article, isn't this the core of the problem? When we've already established that LLM's can do B to A generalizations in-context, why wouldn't they be able to in training?

In one of my experiments I noticed that GPT-4 seems to be perfectly aware of the number of letters in a word in-context, but has difficulty when trained knowledge is involved.

For example it can reliably answer the question:

"Can you tell me how many letters each of the words of the first sentence of our conversation has?"

At the same time it fails with the task:

"Can you rewrite the first sentence of our conversation in a way that preserves its meaning as closely as possible but use only words with an even number of letters?"

It will give an answer but get the letter counts very wrong and it is unable to improve its answer by iteration.

Of course this does not prove that a model cannot be trained to answer tasks involving word length, but GPT-4 seems to have a knowledge gap here (possibly due to tokenization).

Re: Is the reversal curse in LLMs real?

#110

Post by amayne »

user_named wrote: post #93
amayne wrote: post #41

Earlier quoted context omitted.

Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Ma…

1. What do you think the reversal curse implies about LLMs? 2. Do you believe that LLMs are capable of logic? 3. Do you believe that LLMs are intelligent? 4. Do you believe that your blog post shows 3 or 4? If not, what is it about?

1. I don't think the original research paper demonstrated the reversal curse. They claimed that you'd only get random answers from their example prompt. I showed that wasn't the case. I also pointed out what I believe to be a flaw in how they trained their model that when corrected for gave results that were non-random.

2. That depends on what you mean by logic. What would be an example of logical reasoning that would settle this?

3. Alan Turing created the Imitation Game thought experiment to show the futility of this question. If intelligence is something that can be observed and tested, then when we should be able to describe what to test for.

4. I don't make any specific claims about LLMs logic or intelligence. I just wanted to put their claim that LLMs can't generalize from B to A to the test.

Post reply on HN