Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

11–20 of 211 posts

Re: Is the reversal curse in LLMs real?

#11

Post by saithound »

famouswaffles wrote: post #4
ImPostingOnHN wrote: post #3

Earlier quoted context omitted.

It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product i…

It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.

Logician here. We're well aware that "A=B" is not a good formalization of the colloquial/grammarical "A is B", we don't formalize it that way, and we don't regard "if A is B then B is A" is a truth of formal logic.

The people making the argument about reversal curses are not logicians, and most of them don't know anything more about formal logic than what anyone would pick up in an undergraduate "Intro to Proofs" course.

That said, the semantics of the word "is" in natural language really doesn't matter in this debate. The semantics is a red herring, if you will (while a red herring is not semantics).

After all, LLMs cannot learn "the quantity B is mathematically equal to A" from examples of "the quantity A is mathematically equal to B" either, even when the rest of the corpus clearly explains that this _is_ in fact always reversible.

Re: Is the reversal curse in LLMs real?

#12

Post by [deleted] »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

[deleted]

Re: Is the reversal curse in LLMs real?

#14

Post by swatcoder »

ShamelessC wrote: post #10

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel l…

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper.

If you want to defer your own understanding to the author (and overstate their own claims!), that's fine, but a lot of people here are actually more informed about many of the topics and modes of discourse covered in the article and may have something to share with you if you give them credit. And if you're not going to give other commenter's due credit, then you're not really following the spirit of HN. While you can't always know the background of any individual commenter, we have an unusually accomplished and informed community in general and thrive by treating all commenters with the respect we might give the most accomplished and informed -- at least by default. That's why a blanket "RTFA" isn't usually appropriate here.

Re: Is the reversal curse in LLMs real?

#15

Post by earthboundkid »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?

Re: Is the reversal curse in LLMs real?

#16

Post by dj_mc_merlin »

tempestn wrote: post #7

> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them,…

Another similar bug/feature of the brain is how we can immediately know if we like a movie or a book, but if asked what movies or books we like, we often blank or name like 3 pieces. Some information is only designed to be retrieved in certain ways in our brain. The fact that neural networks have similar but different limitations isn't that concerning, just something to keep in mind.

Another funny limitation: when a word is "on the tip of your tongue", what often happened is you thought of it, but your brain rejected it, so now it's on "cooldown" before it appears in your mind again. Usually this mechanism helps but a bug in your brain causes it to be harmful. If you start thinking of something else it "refreshes the cache" and the word comes to you.. we're really not that much less buggy than the machine.

disclaimer: there's other theories about why the phenomenon happens

Re: Is the reversal curse in LLMs real?

#17

Post by famouswaffles »

gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I think a lot of people unwittingly think of the training process as "smart".

Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism.

a "smart" optimizer would look at a reversal applicable sentence and know exactly what bunch of weights to change to store it in such a way as to be recalled reversibly in the future.

Inference may be smart (GPT-4 can reverse in context just fine, potentially play games from only a description) but the training is not.

Re: Is the reversal curse in LLMs real?

#18

Post by Imnimo »

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destroying B is A relationships for weakly represented entities.

Am I missing something here? This paragraph reads like complete gibberish to me.

Also, I don't buy the experiment at the end. If you fine-tune the model with exclusively Tom Cruise data, I want to see proof that it doesn't just answer "Tom Cruise" all the time. I want to see that it says Tom Cruise wrote Aces in the Stream, but doesn't say Tom Cruise wrote Kings in the River.

Re: Is the reversal curse in LLMs real?

#19

Post by famouswaffles »

earthboundkid wrote: post #15
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Ideally, the LLM would be able to tell the is of identity from the is of predication. I found the article a little too defensive. It felt like "of course it doesn't work, there's not enough data!" and okay, sure, but that is a flaw, no?

Well the LLM can tell in that reversal works just fine in-context. This is a recall from training issue and training is dumb.

Re: Is the reversal curse in LLMs real?

#20

Post by thaumasiotes »

akoboldfrying wrote: post #9

Only skimmed this and didn't read the underlying paper, but I was surprised to see no mention of the fact that "A is B" often does not at all imply "B is A" in everyday language: A bird is an animal, but it's wrong to conclude that an arbitrary animal must be a bird.

This is bad news for all the people who like to say "A winner is you!"
Post reply on HN