Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

91–100 of 211 posts

Re: Is the reversal curse in LLMs real?

#91

Post by Applejinx »

This is great work. Andrew's on to something. Thing is, the behavior we see in his form of the LLM is also what we experience as humans.

Good writers, marketers and manipulators know this about people. You seed the ground with the concepts you'll need to introduce, so they're present in the person's 'model' and you can elicit them later, on request.

Formalizing things in the 'reversal curse' manner is like loading the 'mind' of the LLM with habits and assumptions and cutting off its ability to free-associate from seemingly relevant concepts… which is likely to be more valuable in the long run, because an LLM can contain more than a human can. That doesn't mean it will be more intelligent, but it seems reasonable to infer that the LLM can have a broader base of association to draw from, where we as humans tend to be restricted to associations from our own experience. We only get one shot at 'training data', though it's in countless sensory forms, where LLMs are stuck with language as their only window onto experience.

I'm loving the notion of leaving the prompt 'training' stark raving blank. Let's see what comes out of this giant pile of human verbal associations. It is only that, associations, but it's on a grander scale than we're accustomed to. Making it 'answer questions correctly' seems woefully unambitious.

Re: Is the reversal curse in LLMs real?

#92

Post by Grimblewald »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

While red is the apple is not a correct logical statment using conventional understanding of each term, it is still an understandable english sentence, a structure which likely pops up a fair bit. Even if just for scripts of yoda lines. It just goes to show truly how fucky language is and how amazing it is that something even remotely comprehensible comes out of LLMs

Re: Is the reversal curse in LLMs real?

#93

Post by user_named »

amayne wrote: post #41
swatcoder wrote: post #14

Earlier quoted context omitted.

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to…

Hello! I’m the “shark-diving science journalist” in question. First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself. And to clarify a couple things: 1. The shark-diving part is true. 2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Ma…

1. What do you think the reversal curse implies about LLMs? 2. Do you believe that LLMs are capable of logic? 3. Do you believe that LLMs are intelligent? 4. Do you believe that your blog post shows 3 or 4? If not, what is it about?

Re: Is the reversal curse in LLMs real?

#94

Post by bjornsing »

I didn’t read the whole thing, but the first part about Tom Cruise and his mother sounded very flawed: Of course the LLM could learn the reverse relationship if it had better data where the reverse relationship is a common occurrence. The point of the reversal curse argument (I guess) is that it should be able to learn the reverse relationship from entirely other examples [1], but seemingly is not.

1. That is, the LLM should be able to learn that “A is son of B who is female” implies that “B is mother of A”, regardless of who A and B is. It should then be able to apply this pattern to A = “Tom Cruise” and B = “Mary Lee Pfeiffer” and deduce “Mary Lee Pfeiffer is the mother of Tom Cruise” without even a single example.

Re: Is the reversal curse in LLMs real?

#95

Post by TeMPOraL »

zarzavat wrote: post #27

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

Not just languages. It's the case with everything. Easy to spot when you're tutoring someone. You can see they learned "force is mass times acceleration" or "for( ... ) is how you make the same code run multiple times" - they can tell you that when quizzed! But you know they haven't comprehended it until they can reverse it - "I need to compute the mass of the object, which I see accelerating this much under this force; I can pull that from F=ma -> m=F/a!", or "I want to run this code several times, I need a `for` loop!". And getting to that stage is often the longest and most difficult part.

Re: Is the reversal curse in LLMs real?

#97

Post by [deleted] »

maaaaattttt wrote: post #86

The Olaf Scholz exemple from this article is just another exemple of how LLMs can’t count. If you try this prompt: “In this list of words: bike, apple, phone, dirt, tee, sun, glass; which is the fifth word?” it will fail as well. “Fifth” is not connected to any counting ability in LLMs the way it is for us. If you now try this prompt: “Who’s Tom Cruise’s mother in this exemple: “Mary Lee Pfeiffer is Tom Cruise’s moth…

[deleted]

Re: Is the reversal curse in LLMs real?

#98

Post by aade63482 »

godelski wrote: post #61
majormajor wrote: post #22

Earlier quoted context omitted.

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on…

I mean transformers are definitely sequentially biased. There's also human speech bias. But I think it's pretty clear that humans are generally invariant to this kind of prompting as well (given that they have the knowledge. More in a different comment). My surprisal is far higher that the reversal "curse" is considered controversial or even surprising than it was when that sensational tweet dropped. It feels pretty…

[flagged]

Re: Is the reversal curse in LLMs real?

#99

Post by TeMPOraL »

tkgally wrote: post #33
zarzavat wrote: post #27

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

As one gets deeper into learning another language, it’s also important to be aware that the meanings of words in different languages rarely map to each other in clean bijections. Common words especially tend to be semantic clouds, not fixed points of meaning. I don’t know French, but I am sure there are many cases where an English phrase or sentence that includes ‘cat’ should not be translated into French with ‘chat’…

Words are always semantic clouds. Not getting that is what sent philosophy down many a dead end, and the source of plenty of arguments regular people get into all the time. One doesn't need to try mapping between two languages - it's enough problem trying to map within the language itself.

Re: Is the reversal curse in LLMs real?

#100

Post by maaaaattttt »

saliagato wrote: post #88
maaaaattttt wrote: post #86

The Olaf Scholz exemple from this article is just another exemple of how LLMs can’t count. If you try this prompt: “In this list of words: bike, apple, phone, dirt, tee, sun, glass; which is the fifth word?” it will fail as well. “Fifth” is not connected to any counting ability in LLMs the way it is for us. If you now try this prompt: “Who’s Tom Cruise’s mother in this exemple: “Mary Lee Pfeiffer is Tom Cruise’s moth…

It can’t count because what the LLM sees is a bunch of tokens not words

Exactly. And asking what/who/which is the “Ninth” something will always fail (or randomly succeed)
Post reply on HN