Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

71–80 of 177 posts

Re: We Found an Neuron in GPT-2

#71

Post by kilgnad »

PeterisP wrote: post #30
LudwigNagasena wrote: post #13

It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…

I think the lack of explicit linguistic tools is the key to success, forcing/enabling the generic model to learn implicit linguistic tools (there's some research identifying that analysis of specific linguistic phenomena happens at specific places in the NN layers) that work better than what we could implement. "It would be interesting to know how different a model would be if it operated on eg dependency trees inste…

Hmm I think it's sort of a hybrid.

First off, we know that overall the concept of language is humans is an emergent phenomenon. It developed from natural selection from simple components so there's validity that the same thing can occur in an LLM where some overarching emergent structure develops from simple primitives.

At the same time we do know that a sort of universal grammar exists among humans. Our language capacity is biased in a certain way and that it is unlikely for it to learn languages of a very extreme and divergent grammar from the universal one discovered by Noam chomsky. That means our brain is unlikely to be as universally simple as an LLM.

I think the key here is that the human mind has explicit linguistic tools but the these tools are still emergent in nature.

Re: We Found an Neuron in GPT-2

#73

Post by bfung » 

    We found a Neuron in a Neural Network
Nothing new to see here. They pinpoint the nodes where the training bumped up the numbers for one token while not firing other tokens.

Yes, I’m being a bit reductionist, but GPT -> transformer architecture -> neural network. We just have more detailed techniques, a lot more data, storage, and processing power now. But the basics of how a NN works hasn’t changed.

Re: We Found an Neuron in GPT-2

#74

Post by defen »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

What happens if you ask it to complete "I climbed up the pear tree and picked a pear. I climbed up the apple tree and picked a"

Re: We Found an Neuron in GPT-2

#75

Post by clementneo »

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'.

It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1], but how and what LLMs learn exactly remain a black box. When we dig deeper into the mechanisms that drive LLMs, we might get closer to understanding why they work so well in some senses, and why they could be catastrophic in other cases (see: the past month of search-based developments).

I find trying to understand and reverse-engineer LLMs to be a personally exciting endeavour. As LLMs get better in the near future, I sure hope our understanding of them can keep up as well!

[1] https://distill.pub/2020/circuits/zoom-in/

Re: We Found an Neuron in GPT-2

#76

Post by SamBam »

varispeed wrote: post #38
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

My friend who is not a native English speaker told me that one thing he struggled the most while learning English was the "a" and "an". He couldn't grasp the concept how it is possible that person knows which determiner to use before saying the word, until he learned it as just the part of the word and then he uses it depending on the context which one can "feel". So when he sees an apple, he says this is an apple. E…

Are there languages that don't have similar patterns?

Obviously any romance language includes gender, so you need to use the correct gendered article before the noun.

Japanese has different counting words depending on what you're counting.

I don't know if Mandarin has anything similar.

Re: We Found an Neuron in GPT-2

#77

Post by [deleted] »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

[deleted]

Re: We Found an Neuron in GPT-2

#78

Post by eropple »

defen wrote: post #74
josephmiller wrote: post #67

Earlier quoted context omitted.

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

What happens if you ask it to complete "I climbed up the pear tree and picked a pear. I climbed up the apple tree and picked a"

I asked this to ChatGPT.

Its answer?

"Red apple".

...well played.

Re: We Found an Neuron in GPT-2

#79

Post by minkzilla »

IgorPartola wrote: post #63
clementneo wrote: post #53

Earlier quoted context omitted.

The main issue is that GPT is fundamentally an autoregressive language model — it's only predicting the next token based on the prompt at a single time. Every time it wants to predict the next word, it adds the previously predicted word into the prompt, repeating the cycle. We can intuitively guess that the model is 'working out a response that is eventually going to have "apple" in it', but we don't actually know ho…

Do an improv exercise with a friend. Construct a story about a sentient apple one word at a time. I guarantee you that at one point one of you will set up the other with an “an” in your story. This is how I believe this works.

Your friend is thinking ahead that the next word would be apple When they say ‘an’.

But GPT can’t think ahead what token it will add after the one it is on. Or can it? It could “predict” internally the word apple for the next “meaningful” word and output ‘an’ because of this.

Re: We Found an Neuron in GPT-2

#80

Post by SamBam »

Interestingly I feel like humans have this as well, sometimes.

Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like

"the discussion is an... an... epistemological one"

Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to remember the word, but I think sometimes they really don't consciously know what they're going to say, only unconsciously.

It reminds me of a recent New Yorker article about how people think, where the author realized they often have no idea what they're about to say before they open their mouths. [1]

1. https://www.newyorker.com/magazine/2023/01/16/how-should-we-...

Post reply on HN