Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

201–210 of 211 posts

Re: Is the reversal curse in LLMs real?

#201

Post by thfuran »

tysam_and wrote: post #195
thfuran wrote: post #106

Earlier quoted context omitted.

>it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at. But human language is created for and by humans. Is not then operating on language in a categorically different manner an incorrect usage/understanding of language?

No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shanno...

I'm not sure why you think that's a rebuttal.

Re: Is the reversal curse in LLMs real?

#202

Post by tysam_and »

thfuran wrote: post #201
tysam_and wrote: post #195

Earlier quoted context omitted.

No. For more information, please see https://people.math.harvard.edu/~ctm/home/text/others/shanno...

I'm not sure why you think that's a rebuttal.

It is because it is the foundation for the mathematics underpinning your question about communications systems, encodings, and their various structures. It also explains why the choice of the log likelihood is a better fit for estimating a token-based model than some other hand-crafted heuristic.

The version with the Weaver introduction is quite good as well, there are other versions of similar papers covering the topic from different angles, I find it to be well-worth the read.

Re: Is the reversal curse in LLMs real?

#203

Post by mistermann »

bastawhiz wrote: post #177
mistermann wrote: post #166

Earlier quoted context omitted.

You are using your mind to determine what the proper usage is in this specific concrete instance, LLM's do not have your mind to perform that step. LLM's have to work at higher levels of abstraction, and there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. This problem exists for both LLM's and humans, I think it may be…

> there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. I'm not going to hype the technology more than it needs, but it seems to do a pretty damn good job of just this . And to the author's point, it does get this right. And in fact, I'd suspect many of the cases where the LLM gets this wrong are caused by lots of example…

Sure, but here you are describing your subjective[1] opinion on the matter. I do not disagree that this is your opinion, but the fact of the matter is similar but quite different thing.

And this is just one thing that makes sorting out what's going on here difficult - essentially, we have biologically LLM's [thinking they are] decoding silicon LLM's, of course the results are going to be weird and counterintuitive.

[1] Examine the specific language you are using, but also the colloquial ~interpreted/virtualized assertion that one is left with, if they are reading your comment "in good faith".

Re: Is the reversal curse in LLMs real?

#204

Post by mistermann »

thomastjeffery wrote: post #178
mistermann wrote: post #166

Earlier quoted context omitted.

You are using your mind to determine what the proper usage is in this specific concrete instance, LLM's do not have your mind to perform that step. LLM's have to work at higher levels of abstraction, and there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. This problem exists for both LLM's and humans, I think it may be…

LLMs don't derive or follow rules. LLMs don't have anything to do with rules. They implicitly recognize (and model) patterns in text, then continue those patterns. While the patterns that an LLM has modeled may align to the high abstraction we call language, LLMs actually work at a much lower level of abstraction: plain text. It's the content of that text that is significant. We encode language into text, and LLMs in…

You don't actually know what LLM's are "doing", comprehensively. You are describing your map/intuition of what is going on - it is a cultural convention (and a well enforced one at that), so an easy mistake to make.

Re: Is the reversal curse in LLMs real?

#205

Post by mistermann »

naasking wrote: post #179
mistermann wrote: post #166

Earlier quoted context omitted.

You are using your mind to determine what the proper usage is in this specific concrete instance, LLM's do not have your mind to perform that step. LLM's have to work at higher levels of abstraction, and there the multiple meanings of "is" (which is also very commonly used to mean it is my opinion that X "is" Y) makes sorting things out very difficult. This problem exists for both LLM's and humans, I think it may be…

LLMs effectively disambiguate in an arguably similar way humans would: look at the same term in multiple contexts to reduce confidence in incorrect interpretations, leaving only correct interpretations.

> leaving only correct interpretations

Unlike humans, LLM's can consistently realize (or at least claim to realize) when they've made a mistake (like assuming one's beliefs are necessarily correct, [because some persuasive story]).

Re: Is the reversal curse in LLMs real?

#206

Post by godelski »

famouswaffles wrote: post #199
godelski wrote: post #193

Earlier quoted context omitted.

Sure, definitely not "free" and why I say nearly invariant. But that is also a different condition that what I was attempting to express. What I was trying to say is under a condition where you already know 고양이 means cat, then the reverse should not be too difficult (though certainly not completely invariant as all language learners know that made up silly phrase half direct translation thing). Conditions I mean: 고양이…

If you haven't learned Hangul, then you can learn it from this playlist - https://www.youtube.com/watch?v=Z9ZxsgMAZmI&list=PLbFrQnW0BN... If you have learned Hangul, then you can start with Lingodeer(app), TTMIK(talk to me in Korean)’s Grammar books, or Billie's beginner playlist - https://www.youtube.com/watch?v=sx0yyQqkpqo&list=PLbFrQnW0BN... . These are all extremely beginner friendly. You can just look at the opt…

Wow this was an incredible response. I don't even know how to properly respond to it but I've favorited it and bookmarked it. Thanks a ton!!!!

Re: Is the reversal curse in LLMs real?

#207

Post by mistermann »

croes wrote: post #191
mistermann wrote: post #170

Earlier quoted context omitted.

It only takes two humans getting it correct once for your statement to be technically true though, demonstrating how easily seemingly innocuous and straightforward language can be misleading. The opposite statement, humans don't know when "is" means equal is also a true statement, perhaps even more true.

But you wouldn't think of the latter group of humans as the next big thing for many important tasks. I doubt MS would put their service in lot of their products.

> But you wouldn't think of the latter group of humans as the next big thing for many important tasks.

I consider them the target audience for upgrades, I think the ROI could be massive.

Re: Is the reversal curse in LLMs real?

#208

Post by foobarqux »

famouswaffles wrote: post #197
foobarqux wrote: post #28

Earlier quoted context omitted.

But that's the problem isn't it? In some cases they are equivalent and in some cases they are not and a next-word-predictor needs to have "explicit" training data (i.e. it is not doing "reasoning") whereas a human can infer. This isn't surprising if you think about how the machine actually works rather than treating it like a sacred magic box. The default assumption for why there is any "success" for in-context learn…

LLMs can already reverse logical implications. Literally this is not a problem in context. The model can infere all this just fine. Even the original paper makes this clear. This is a recall from training problem, not a logical inference one.

I don't understand what you are saying. LLMs can't generally reverse implications, that's what the paper says. This in turn suggests that what you are observing in-context is some other statistical heuristic, not a "logical inference capability".

I don't know in what way this is some trivial "recall" problem as you suggest. Does the reversed implication in question need to be in the training data explicitly or not? If it does I don't understand how you can claim a logical inference capability.

Re: Is the reversal curse in LLMs real?

#209

Post by famouswaffles »

foobarqux wrote: post #208
famouswaffles wrote: post #197

Earlier quoted context omitted.

LLMs can already reverse logical implications. Literally this is not a problem in context. The model can infere all this just fine. Even the original paper makes this clear. This is a recall from training problem, not a logical inference one.

I don't understand what you are saying. LLMs can't generally reverse implications, that's what the paper says. This in turn suggests that what you are observing in-context is some other statistical heuristic, not a "logical inference capability". I don't know in what way this is some trivial "recall" problem as you suggest. Does the reversed implication in question need to be in the training data explicitly or not? I…

>LLMs can't generally reverse implications, that's what the paper says

No that's not what the paper says. It says training won't immediately store this information in a way as to make it reversible.

Let's get one thing straight. This is a common problem for human learning as well. Anyone who has used Anki for language learning will tell you that if you just train on target language word on the front and native language word on the back, you will fail the reverse unless you specifically train for that.

This is specifically a problem of recall. Not a problem of making the logical inference. If you ask the human language learner immediately he has learnt the new word for the reverse direction, he will obviously tell you the correct word.

But later he may not recall the reverse even if he remembers the original. Again this is a problem of recall rather than the ability to make logical inferences.

In the same vein, if you give the LLM the original direction in context and immediately ask the reverse, it will correctly tell you. It can make the logical inference.

Re: Is the reversal curse in LLMs real?

#210

Post by foobarqux »

famouswaffles wrote: post #209
foobarqux wrote: post #208

Earlier quoted context omitted.

I don't understand what you are saying. LLMs can't generally reverse implications, that's what the paper says. This in turn suggests that what you are observing in-context is some other statistical heuristic, not a "logical inference capability". I don't know in what way this is some trivial "recall" problem as you suggest. Does the reversed implication in question need to be in the training data explicitly or not? I…

>LLMs can't generally reverse implications, that's what the paper says No that's not what the paper says. It says training won't immediately store this information in a way as to make it reversible. Let's get one thing straight. This is a common problem for human learning as well. Anyone who has used Anki for language learning will tell you that if you just train on target language word on the front and native langua…

Humans have reasoning capabilities that are separate from memory. If you want to show that LLMs have reasoning capability you try to create carefully constructed adversarial experiments that try to distinguish between that hypothesis and others, you do not just show that there are cases that favor your hypothesis.

And although it's a complete waste of time because the answer is obvious when you do do these types of adversarial experiments, like adversarial blockworld, you find that LLMs are not internalizing abstract principles inherent to reasoning and are instead doing something like string-pattern-patching.

Post reply on HN