Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is the reversal curse in LLMs real?

andrewmayne.com

Post reply on HN
Print view View original

41–50 of 211 posts

Re: Is the reversal curse in LLMs real?

#41

Post by amayne »

swatcoder wrote: post #14
ShamelessC wrote: post #10

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel l…

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to…

Hello!

I’m the “shark-diving science journalist” in question.

First of all, you can run the experiments like I did and test this yourself. I’m not asking anyone to take my word. Just do what I did: Read the original paper. Test the claims for yourself.

And to clarify a couple things:

1. The shark-diving part is true.

2. I’ve never been a journalist of any kind that I’m aware of unless you count writing for Skeptic Magazine. I’ve had many, many jobs though.

3. I started at OpenAI as a software engineer and member of technical staff. When I started there was just over a hundred people there. The lines between engineering were and are blurry.

4. I was the original prompt engineer at OpenAI and discovered many of the examples for using GPT-3 and wrote a lot of the original documentation. Internally my title was “prompt whisperer.”

5. I’m in the GPT-4 research paper for my contributions to model capability. I helped find abilities for long-text, vision, etc.

6. I was given the title Science Communicator when I started doing background briefings for media, etc., but still worked on model capability and other things.

7. I left OpenAI two months ago to work on a startup.

Best,

Andrew Mayne

Re: Is the reversal curse in LLMs real?

#43

Post by tikimcfee »

emme wrote: post #40
tysam_and wrote: post #38

Earlier quoted context omitted.

Bit of a nitpick, "red is the apple" is both a valid sentence and one that also conveys the original relationship held by the opposite phrasing, so, in this case, "the apple is red" does indeed imply "red is the apple", and accurately so.

A proper example is probably "an apple is red" and "red is an apple".

What you’re missing that is “red is an apple” is also possibly saying in a metaphorical sense that red is like an apple to someone - delicious, a treat, perhaps otherwise representative. In that way, the encoding of “is”’ is exactly correct - it’s an ordered pair of glyphs that imply a weak form of assignment or description. : apologies, replied to the wrong post, meant to push this up one.

Re: Is the reversal curse in LLMs real?

#44

Post by [deleted] »

swatcoder wrote: post #14
ShamelessC wrote: post #10

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel l…

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to…

[deleted]

Re: Is the reversal curse in LLMs real?

#45

Post by ShamelessC »

swatcoder wrote: post #14
ShamelessC wrote: post #10

The article is a decent peer review and refutation of “the reversal curse”. Some of the comments given here clearly haven’t read the whole article though - arriving at similarly skeptical conclusions that are clearly present and expanded on in the article. Why do people feel the need to do this here? Armchair commentary on advanced material is one of the main reasons I avoid Reddit. And furthermore why does it feel l…

The author is a shark-diving science journalist who's been paid a lot of money by OpenAI, not a researcher, and the article is neither a "peer review" or an expert "refutation". It's a meandering and sometimes thought-provoking exploration of how LLM's can be coaxed to deliver on statements sort of like (but not actually equivalent to) the ones that failed in the paper. If you want to defer your own understanding to…

All due respect, when I wrote this there were about 5 comments that were 100% not from those who are "more informed" than the author (and perhaps were even less informed than myself, although I don't claim to be an LLM researcher or anything more than an interested internet user).

Just the same it's been my experience, ESPECIALLY in machine learning, that users on HN are consistently over confident, bad at making future predictions, and generally just easily excitable. Does that mean I'm talking about you or commenters you like? Probably not! You seem relatively informed (although I'm not fond of you attacking the author of the article for things it isn't claiming - something truly not in the spirit of HN).

Re: Is the reversal curse in LLMs real?

#46

Post by thehappypm »

dj_mc_merlin wrote: post #16
tempestn wrote: post #7

> This isn’t a failure of neural networks. It’s a feature. It’s why you’re not flooded with every single memory and experience you’ve ever had every moment. This is an interesting point, and made me think on whether this "reversal curse" is something we experience with our own, human neural networks. I think it is. Like, I can imagine being given a character in a movie, being able to tell you what actor played them,…

Another similar bug/feature of the brain is how we can immediately know if we like a movie or a book, but if asked what movies or books we like, we often blank or name like 3 pieces. Some information is only designed to be retrieved in certain ways in our brain. The fact that neural networks have similar but different limitations isn't that concerning, just something to keep in mind. Another funny limitation: when a…

Interestingly, I think the “favorite movies” scenario is also an artifact of our training dataset (our experiences). You spend hours watching a movie, so you have a lot of data about it, and have a lot to build an opinion. But comparing movies? Not something we think about for hours and hours on end. But, people who are movie critics or movie buffs train themselves to do it.

Re: Is the reversal curse in LLMs real?

#47

Post by cryptoz »

tikimcfee wrote: post #43
emme wrote: post #40

Earlier quoted context omitted.

A proper example is probably "an apple is red" and "red is an apple".

What you’re missing that is “red is an apple” is also possibly saying in a metaphorical sense that red is like an apple to someone - delicious, a treat, perhaps otherwise representative. In that way, the encoding of “is”’ is exactly correct - it’s an ordered pair of glyphs that imply a weak form of assignment or description. : apologies, replied to the wrong post, meant to push this up one.

The correct sentence would start with a capital letter: “Red is an apple.” This is also completely valid as in a cartoon character of an apple named Red. The subject being the start of the sentence compounds the uncertainty in meaning.

Re: Is the reversal curse in LLMs real?

#48

Post by amayne »

Imnimo wrote: post #18

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destro…

I've added your suggested test to the blog post: https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea...

Spoiler: It doesn't say "Tom Cruise."

Re: Is the reversal curse in LLMs real?

#49

Post by tysam_and »

famouswaffles wrote: post #17
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I think a lot of people unwittingly think of the training process as "smart". Similar criticisms are "well there are many descriptions of game x it would have read so why doesn't it play x well". But gradient descent is a dumb optimizer. Training itself is not actually like someone reading a text anymore than evolution is like someone thinking about the best way to augment an organism. a "smart" optimizer would look…

I'm not really sure if I understand the intuition here, this seems rather disconnected from what I understand to be the math of optimization.

It seems like you're referring to an associative Hebbian/Hopfield-like lookup, which the current 'dumb' optimizers already do. Better yet, the learning rates are normalized by the diagonal of the empirical Fisher so that the learning w.r.t. to some estimated expected information is more constant, for said associative lookup operation.

Additionally, the training loop (which you call 'dumb') is just...a teacher-forced version of inference, which you call 'smart'?

It's better to simply minimize the log-likelihood in a scalable way. Hand-engineered solutions rarely survive compared to strong-scaling ones.

Re: Is the reversal curse in LLMs real?

#50

Post by tysam_and »

majormajor wrote: post #22
gipp wrote: post #8

I fall somewhere on the skeptic side of the LLM spectrum. But this "flaw" just does not seem to have the force that its proponents seem to think it does, unless I'm missing something significant. Simply because in the context of natural language (Rather than formal logical statements), "A is B" does not imply "B is A" in the first place. "Is" can encompass a wide variety of logical relationships in colloquial usage,…

I don't think this is the right explanation, or really that relevant, the suggestions from 'og_kalu and in the article sound more accurate to me. It seems like understanding when "is" is reversible is pretty core to the capabilities of the model, but that's different than having a lot of facts memorized. For instance, a model should be able to answer "who is the star of Mission Impossible" with "Tom Cruise" based on…

It's a bit of mathematical bikeshedding, hardcoding reversability would cause far more problems than it would help.

Best to simply scale log-likelihood-based training, next-token-based training trivially contains a requirement for learning all of the subproblems that predict said, next token, and hardcoding something to get warm human fuzzies would be creating a biased estimator (and move us back towards the 90s a bit).

Models already very constantly do context-dependent token utilization, it's an autoregressive feature based on the entire stream of incoming tokens. Humans have a bias to focus on the 'last token used', this is not what language models look at.

Post reply on HN