Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

161–170 of 177 posts

Re: We Found an Neuron in GPT-2

#161

Post by theGnuMe »

HarHarVeryFunny wrote: post #159
psychphysic wrote: post #158

Earlier quoted context omitted.

Its chosen by the ngram and randomly so, that does suggest it is completing the text a word at a time.

Did you check the Vonnegut writing rules example I posted at top of this thread - in particular look at Bing/GPT's explanation of how its cake story matches up to Vonnegut's rules ? It's hard to imagine how it could have come up with such a coherent story, checking all the rules, if it was only conceiving of it's continuing story on a word by word basis. It's not as if sentence #1 matches rule number 1, sentence 2 ma…

Kurt Vonnegut is a conditional sub space of the embedding vectors.

Re: We Found an Neuron in GPT-2

#162

Post by bilsbie »

clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

Near work! I’m still confused how it knows to use “an” if it hasn’t chosen the word after it yet?

Re: We Found an Neuron in GPT-2

#163

Post by bilsbie »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

As a test, during text generation could you change an “a” to an “an” and see if it changes the noun. Or did it already have a noun in mind and it sticks with that.

Re: We Found an Neuron in GPT-2

#164

Post by solarmist »

kilgnad wrote: post #86
vajrabum wrote: post #82

Earlier quoted context omitted.

Lots of linguistics researchers would disagree about the existence and necessity of universal grammar or Chomsky's 'language acquisition device' and in fact the success of LLMs and statistical models which very clearly have no LAD and no universal grammar over feature engineering or grammar based schemes suggests the opposite.

I'm outdated on this research. I took a linguistics class in UCLA about a decade ago and at the time universal grammar was the prevailing norm. Has it changed since then? Have they found a significant number of human civilizations that use divergent grammars?

Chomsky’s universal grammar is a modification of Harris’s Operator grammar.

Harris’s operator grammar was based on set theory however Chomsky was enamored with formal logic and ran in that direction. Also, Chomsky became famous while Harris didn’t.

Operator grammar is self-discoverable and he published a full description of English grammar in the 1980’s using this theory and an extension in the 1990’s which generalized to other languages.

It is not fully deterministic (final word selection and ordering is probabilistic), but it is much more convincing to me and in line with how we understand brains to work.

Over the last 20 years Chomsky has begun to fall out of favor because it’s just so complicated and requires external structures, etc.

Re: We Found an Neuron in GPT-2

#165

Post by solarmist »

Baeocystin wrote: post #58
LudwigNagasena wrote: post #13

It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…

Approaches such as you describe have been the dominant method for decades. That we finally 'cracked' natural language generation with tools that literally encode nothing about grammar ahead of time is one hell of a lesson, early days as it is in the learning of it.

Harris, Chomsky’s advisor, published his operator grammar over the decades and one of the key features was that it is completely self-discoverable.

Probability and observation are all that is required to understand a language.

Re: We Found an Neuron in GPT-2

#166

Post by tqi »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

Thanks for responding, and for the cool article! Agree this is pretty tangential to the rest of the article.

So (again, I am very much not an expert so please correct me if I'm wrong) I guess my analogy would be if instead of predicting one word at a time, it predicted one letter at a time. At some point, would only be one word that could fit. So if prompted with "SENTE", and it returned "N", that doesn't mean that it's thinking ahead to the "CE" / knows that it is spelling "SENTENCE" already.

Is that a correct way to think of it?

Re: We Found an Neuron in GPT-2

#167

Post by sharemywin »

clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

you might find this paper interesting:

https://arxiv.org/abs/2202.05262

Locating and Editing Factual Associations in GPT

Re: We Found an Neuron in GPT-2

#168

Post by sharemywin »

HarHarVeryFunny wrote: post #157
gbasin wrote: post #155

Earlier quoted context omitted.

There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.

Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").

I see 2 options:

1. we don't know what they(coding layer between bing and GPT) look up and store as a prompt aka working memory.

2. it can do the equivalent of receiving it's own prompt silently.

I seen with code it outputs the step for the code then writes the code.

so there's some kind of plan and execute going on. maybe it can do that in model some how

Re: We Found an Neuron in GPT-2

#169

Post by dpaleka »

sharemywin wrote: post #167
clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

you might find this paper interesting: https://arxiv.org/abs/2202.05262 Locating and Editing Factual Associations in GPT

That paper (ROME) was the most famous paper in the field last year :)

See also new interesting developments breaking the connection between "Locating" and "Editing":

https://arxiv.org/abs/2301.04213

Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models

Re: We Found an Neuron in GPT-2

#170

Post by hackinthebochs »

sharemywin wrote: post #168
HarHarVeryFunny wrote: post #157

Earlier quoted context omitted.

Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").

I see 2 options: 1. we don't know what they(coding layer between bing and GPT) look up and store as a prompt aka working memory. 2. it can do the equivalent of receiving it's own prompt silently. I seen with code it outputs the step for the code then writes the code. so there's some kind of plan and execute going on. maybe it can do that in model some how

>so there's some kind of plan and execute going on. maybe it can do that in model some how

The simple answer is that the internal state that picks the next token is stable over iterations so that the model can follow a consistent plan over multiple token outputs. Then as the plan "unfolds" in the output tokens, these tokens help stabilize the plan further, thus creating consistency over long generations.

Post reply on HN