Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

51–60 of 211 posts

Re: Is the reversal curse in LLMs real?

#51

Post by tysam_and »

cryptoz wrote: post #47
tikimcfee wrote: post #43

Earlier quoted context omitted.

What you’re missing that is “red is an apple” is also possibly saying in a metaphorical sense that red is like an apple to someone - delicious, a treat, perhaps otherwise representative. In that way, the encoding of “is”’ is exactly correct - it’s an ordered pair of glyphs that imply a weak form of assignment or description. : apologies, replied to the wrong post, meant to push this up one.

The correct sentence would start with a capital letter: “Red is an apple.” This is also completely valid as in a cartoon character of an apple named Red. The subject being the start of the sentence compounds the uncertainty in meaning.

Yes, this is because there's no assignment happening up front w.r.t. the word 'red'. However, you'll notice the same kind of ambiguity for the word 'apple' in the reversed sentence, as the word 'The' can imply quite a few things following it.

The entropy gotta get slung around somehow.

Re: Is the reversal curse in LLMs real?

#52

Post by [deleted] »

tkgally wrote: post #33
zarzavat wrote: post #27

Humans are also vulnerable to the reversal curse! When you learn languages you have to learn both directions (chat is cat and cat is chat), anybody who has built an anki deck will know this, otherwise you will be better in one direction than the other.

As one gets deeper into learning another language, it’s also important to be aware that the meanings of words in different languages rarely map to each other in clean bijections. Common words especially tend to be semantic clouds, not fixed points of meaning. I don’t know French, but I am sure there are many cases where an English phrase or sentence that includes ‘cat’ should not be translated into French with ‘chat’…

[deleted]

Re: Is the reversal curse in LLMs real?

#54

Post by famouswaffles »

tysam_and wrote: post #49
famouswaffles wrote: post #17

Earlier quoted context omitted.

I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look…

I'm not really sure if I understand the intuition here, this seems rather disconnected from what I understand to be the math of optimization. It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected informatio…

smart is in quotes here for a reason. It's 'dumb' in relation to many people's expectations.

>Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'?

What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. There are qualitative differences. The same models have no problem with this 'curse' when the information is provided in context for example.

>It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones.

I never said anything about it being bad.

Re: Is the reversal curse in LLMs real?

#55

Post by rossdavidh »

famouswaffles wrote: post #4
ImPostingOnHN wrote: post #3

Earlier quoted context omitted.

It doesn't seem obvious at all unless you start with the assumption that not being able to derive A=B -> B=A is "obvious", which it clearly isn't to most people. Indeed, just because the lack of that capability is a result of the design of LLMs doesn't mean it's a feature of LLMs. It could also be that it's a bug of LLMs. Which one depends on what the expected behavior is, and the expected behavior from the product i…

It's "obvious" because A is B -> B is A is not a thing that is actually true for the vast majority of text (or really any kind) constructions. It's only a truth of formal logic.

It is a thing that nearly any non-technical user of LLM's would expect it to be able to do, and be surprised at it not doing. Which is a bug, if you expect it to be a service/piece of software that someone not trained on using LLM's to be able to do. The author more or less admits this in the section entitled "Model training is a dark art"

'I’ve been playing around with fine-tuning LLM models for years and still don’t have any hard and fast one-size-fits-all rules to apply. Every dataset lends itself to a specific way of training.'

But this is precisely the reason why nearly every claim about what "AI" will soon be able to do, is misguided. The failures of LLM's are exceedingly unintuitive to anyone who doesn't have a lot of experience with them (and maybe sometimes to people who do).

Spreadsheets can be used by people who don't know how spreadsheets' internals work; after a bit of training (in my experience, about ten minutes) they can get a decent intuition about how to use a spreadsheet (at least for the simple stuff). The same is true of well designed web pages, word processors, music players, etc. Even software with more complex training requirements like CAD, statistics packages, video editing, etc. will usually behave in a more-or-less intuitive fashion for a user of the appropriate target group.

LLM's fail in unexpected (and sometimes hard to spot) ways, and are being marketed as if they are a tool for the general population to use, when their training (and failure modes) are still a "dark art" even for people with years of experience in them. That is not a feature.

Re: Is the reversal curse in LLMs real?

#56

Post by tysam_and »

famouswaffles wrote: post #54
tysam_and wrote: post #49

Earlier quoted context omitted.

I'm not really sure if I understand the intuition here, this seems rather disconnected from what I understand to be the math of optimization. It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected informatio…

smart is in quotes here for a reason. It's 'dumb' in relation to many people's expectations. >Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'? What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training. There are qualitative differences.…

> What powers In context learning is not very well understood but it doesn't appear to be or really work exactly like just a non teacher-forced version of training.

I mean, yes. One distills information from a training set into a compressed representation, and the other generates a compressed representation that yields (more or less) fixed state space attractors. It's just inducing a bias over the state space of the network, nothing incredibly special, though I'd consider the initial stage of training to be the most important, as it is responsible for all of the ingest of all of the embedded information the network will be using during autoregressive inference (especially w.r.t. the context of doing so during longer generations).

So the notion of 'in context learning' is one I find to be a bit of an illusion, of course, as no actual learning is being done, just the induction of biases, which appears to give rise to a transiently-'better trained' network.

You could see this as a bit straightforward perhaps, but I feel it needs to be said.

Re: Is the reversal curse in LLMs real?

#57

Post by cratermoon »

xelxebar wrote: post #36
cratermoon wrote: post #31

If I'm reading this correctly, the author is saying that it's not a failure of logical deduction if the training data doesn't include the reversal. In other words, he's saying that if the data contains "Tom Cruise is the son of Mary Lee Pfeiffer” but not “Tom Cruise’s mother is Mary Lee Pfeiffer”, then the model's inability to determine the latter is "an explanation of how neural networks function than a model’s inab…

Did you see the end of the article, where the author uses a small example and gets "B is A" generalization? These are the salient takeaways I got: - Is/Was wording might matter. This is something probably a bug. - 30 facts about a person might simply be too little for "B to A" generalization - Extra precision/context in the prompt can help locate the "B to A" inference. - How you cut up your training data can bias in…

OK, but how does that not just demonstrate LLMs can't generalize absent massaged training data?

> it's not as stable as you'd want.

Which I take to mean that a model can sometimes confabulate "B is A", solely out of random variation, and that it's possible to bias the data and prompt to generate the expected response. The model hasn't done any logical deduction, the response is just a bias-influenced lucky break.

Re: Is the reversal curse in LLMs real?

#58

Post by dr_dshiv »

> So in summation: I don’t think any of the examples the authors provided are proof of a Reversal Curse and we haven’t observed a “failure of logical deduction.” Simpler explanations are more explanatory: imprecise prompts, underrepresented data and fine-tuning errors.

Phew!

Re: Is the reversal curse in LLMs real?

#59

Post by godelski »

I didn't know anyone was even suspicious of the reversal "curse". I thought it went viral more because people were surprised anyone was surprised and the tweet was highly sensationalized. It feels more like the __expected__ results considering both speech patter bias __and__ that "causal" attention is sequentially biased. Hell, we saw the same things in RNNs, we see it in classic autoregressive models, and so on. But it definitely isn't how humans encode information because if you had all 3 pieces of knowledge (you know who Tom Cruse is, you know who Mary Pfieffer is, and you know the relation between Mary and Tom is mom/son) then your ability to recall is (nearly) invariant to the ordering. Hell, we're so robust you can caveman speak like "Mary Pfieffer son who!" and still get the answer or caveman yoda speak "Who Mary Pfieffer son is?" Some languages even have these orderings but we're impressively robust (enough that I think we trick ourselves a lot when subtlety comes into play. AKA overfitting).

So I find it weird to call this "a feature" and also act like this is surprising.

But can I also take a minute to just say I really hate GPT experiments? They're performed on a stochastic model, with proprietary weights, proprietary training data, proprietary training methods, and above all is constantly changing and at a rather fast pace. It makes for a very non-scientific process as you can't decouple a lot of important factors and reproduction is a crap shoot. It is not a very good way to go about studying "how __LLMs__ work" and is rather "how does GPT work at this particular moment in time and aggregating all these unknowns?" There's some bitter sweetness because I do like that GPT is free but it feels like a major edge that they have is that the community just does a lot of free research for them and in ways where they can better interpret results than the people who performed the experiments in the first place. I really do believe that academic works shouldn't be focusing on proprietary and dynamic methods. It's fine to include them in results (and preferably added post review to avoid identity spoilage (or we openly acknowledge that double blind is a joke)) but I'd rather most researchers focusing on the general concepts and with the ability to dive down the rabbit hole than playing a wack-a-mole game.

Also, I'd totally love it if more research papers were instead blog posts. Kudos to anyone posting their research on blogs, academic or not (I don't care about your creds, your real creds are your work). Papers are about communicating with fellow scientists, right? Why do we need journals and conferences these days?

Re: Is the reversal curse in LLMs real?

#60

Post by eru »

tysam_and wrote: post #38
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

Perhaps have a look at this famous syllogism instead:

(1) Mortal was Socrates.

(2) All humans are mortal.

(3) Therefor all humans are Socrates.

Post reply on HN