Isn't this solvable by creating training data that says B is A? Instead of always training it on A is B.
Is the reversal curse in LLMs real?
121–130 of 211 posts
Re: Is the reversal curse in LLMs real?
#122I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel…
your first two links are duplicates
Re: Is the reversal curse in LLMs real?
#123>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destro…
I've added your suggested test to the blog post: https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea... Spoiler: It doesn't say "Tom Cruise."
In one of your reproductions of the results from the paper it would spit out random other names from the dataset. But from your description of the Tom Cruise dataset, there were no other names included. Why didn't you give it the possibility to fail in the same way?
Re: Is the reversal curse in LLMs real?
#124I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel…
> Even when allowed to surf the web, ChatGPT gets it wrong: > Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Gerhard Schröder Its probably counting Walter Scheel, who was Acting Chancellor in 1974, and you are probably not. When I asked ChatGPT to list the chancellors in order and identify the eighth, it listed eight ending in Schröder, with Scheel and his ten day Acting Chancellor tenure…
If you think ChatGPT is "counting" anything, you have massively missed the point.
Re: Is the reversal curse in LLMs real?
#125> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them,…
Another similar bug/feature of the brain is how we can immediately know if we like a movie or a book, but if asked what movies or books we like, we often blank or name like 3 pieces. Some information is only designed to be retrieved in certain ways in our brain. The fact that neural networks have similar but different limitations isn't that concerning, just something to keep in mind. Another funny limitation: when a…
Yeah this becomes very noticeable when you're deliberately learning facts with Anki.
For example when trying to learn immunology, "This bacterium causes neural bacteriumeritis", score 100% on the set of cards after a few sessions.
Add "What are the possible causes of neural bacteriumeritis?" and sometimes I'd draw a blank.
There's a lot of nuance in how we learn facts and their relationships.
It reminds me a lot of data structures in programming to be honest. It's a lot easier to retrieve a person's age from a dictionary indexed by first-name+last-name than from a list with 10000 objects with people in them.
Re: Is the reversal curse in LLMs real?
#126I played along the example of the chancellor question. If you deviate a little from the examples in the article GPT-4 gets it all wrong: Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) https://chat.openai.com/share/937795ea-bd91-43ee-bd76-1a125f... Who was the eighth Federal Chancellor of the Federal Republic of Germany? - Helmut Kohl (wrong, Angela Merkel…
>Who is the eighth Federal Chancellor of the Federal Republic of Germany? - Olaf Scholz (wrong, Angela Merkel) Technically there's no right answer to this question. "Is" implies present. But Angela Merkel isn't the present chancellor. Olaf Scholz is. But he's not the eigth.
Technically, Angela Merkel never stopped being the person who served as the eighth chancellor.
Re: Is the reversal curse in LLMs real?
#127I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…
This is approximately where I wound up. I had an initial knee-jerk reaction after reading the paper, but your kind of reasoning brought me back down to earth.
I don't know that I even want these things to generalize. I get it in academic terms, but from a CTO perspective how much this matters to the business? We already know our policies and procedures and aren't exactly excited about the idea of something getting curious with flipping things around.
I don't want it to generalize and start making weird assumptions like "some men are tall". I want it to say "I don't know", or provide a statistical signal indicating the same.
Re: Is the reversal curse in LLMs real?
#128Earlier quoted context omitted.
This simply isn't true of humans for the kinds of examples used in the paper. If I read "Olaf Scholz was the ninth Chancellor of Germany" then I have no trouble reversing that and determining that the 9th chancellor of Germany was Olaf Scholz. This is not an inference that's 'cognitively challenging'. Humans may sometimes fail to make simple inferences of this sort when building their factual databases, but they don'…
> This simply isn't true of humans for the kinds of examples used in the paper. If I read "Olaf Scholz was the ninth Chancellor of Germany" then I have no trouble reversing that and determining that the 9th chancellor of Germany was Olaf Scholz. This is not an inference that's 'cognitively challenging'. That's in-context though. LLMs don't fail in-context either. For a better comparison, recall your school experience…
>For a better comparison, recall your school experience, say with history lessons, or geography lessons - where you would cram a hundred "A is B" relationships, and then take a test that demanded you know the reversals. Not as easy.
This doesn't correspond to my school experience. In general I don't feel that I have to separately memorise "A is B" and "B is A". For example, if I learn that Elizabeth I was Henry VIII's daughter, I don't also have to learn that he was her father.
>LLMs can perform logical deductions, but at no point in the training process is this capability used.
That is exactly what the paper says.
Re: Is the reversal curse in LLMs real?
#129Earlier quoted context omitted.
Nah. The context of all of this is people claiming LLMs are AI. If so they would be able to reverse A is B, and also know when not to, so your example is irrelevant. The paper shows that it cannot reverse. Then the blog posts goes around different disingenuous ways of making excuses for why it can't reverse at the same time as trying to show that it can reverse in the Tom Cruise training example (extremely disingenuo…
I still call bullshit on this. Humans don't do reversals for free either. We may memorize the reversal, or first, infer it . That's an extra step, often very cognitively challenging. I'd like to see experiments showing how "reversal curse" fares when LLM is allowed to make the inference. Like: "Please complete the sentence: [your B->A] - but reason through the process first. Start with identifying the subject, then w…
Contrary what you claim, there is nothing in common between LLMs and humans. We are intelligent and conscious, these dumb models are not.
Re: Is the reversal curse in LLMs real?
#130I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…
This is approximately where I wound up. I had an initial knee-jerk reaction after reading the paper, but your kind of reasoning brought me back down to earth. I don't know that I even want these things to generalize. I get it in academic terms, but from a CTO perspective how much this matters to the business? We already know our policies and procedures and aren't exactly excited about the idea of something getting cu…
Then you don't want an LLM you want something deterministic