Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

121–130 of 177 posts

Re: We Found an Neuron in GPT-2

#121

Post by FeepingCreature »

vasco wrote: post #114
FeepingCreature wrote: post #111

Earlier quoted context omitted.

I usually know what I'm about to actually say very shortly before I say it. This has occasionally led to emergency course corrections. But I do sense the "shape" of a sentence well before I say it. I think there's something like a stage design in the brain: - symbolic or model deliberation - verbal expression - vocalization And each of those stages can be consciously introspected on, but people will naturally develop…

I don't think you are right. I experience the same feeling of focusing on an idea but not having a fixed idea what exactly I'll say, but I can also prepare full sentences if I want to. It's just most of the time I make an effort to put my brain in speech-autopilot mode. I think in fact it's harder to let yourself be lead by it without consciously introspecting, at least I find I'm able to discuss way quicker as I can…

I don't think it's like ... if an idea gets moved to the verbalization stage, that you're compelled to say it. It's more that some people seem to have little control over what happens at that stage, ie. they can operate on a concept but they can't predict the way it'll be spoken.

Like

[Conceptual stage] --??-- consciousness

V

[Verbalization stage] --??-- consciousness

V

[Vocalization]

So if you don't have conscious access to verbalization, you only realize how a thought "will sound" after you say it. Conversely, if you don't have conscious access to conceptualization, you end up thinking that "thinking" always involves "thinking out loud", because "thinking out loud" (verbalizing) is the only way you have to query your conceptual layer. You literally only become aware of your own thinking after the thought is already pretty far along. At the extreme, you can have conscious access to neither stage and require vocalization to reflect on your own thinking.

Re: We Found an Neuron in GPT-2

#122

Post by vore »

nerdponx wrote: post #93
hgsgm wrote: post #48

Earlier quoted context omitted.

Surely "an eagle" is [ an][ eagle] since the an starts a new word.

The OpenAI tokenizer tool agrees with you, unless it's at the start of the document. e.g. "An eagle flew into its nest ..."

Yup, if placed at the start of the document, "An eagle [...]" would be [2025, 31176] and "Anoxic [...]" would be [2025, 18047].

Re: We Found an Neuron in GPT-2

#123

Post by vore »

phkahler wrote: post #57
vore wrote: post #34

Earlier quoted context omitted.

It goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not…

So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? How is it "clocked" to get successive tokens?

GPT is autoregressive, so it uses the previously generated output to produce the next prediction.

Re: We Found an Neuron in GPT-2

#124

Post by redox99 »

EricLeer wrote: post #120
redox99 wrote: post #119

Earlier quoted context omitted.

Its input is a sequence of tokens (so text), and its output a list of probabilities for the single next token. You pick the highest probability token, append it to input and execute the neural network again. If it outputs an " " token, it means it's done you stop this "while loop". In other words, it outputs one token at a time.

True, but this entire loop happens within the model, if you would return an output at every intermediate step the model would be extremely slow. My take on why they have build the output layer like it is, is that next to feeling more human, it also forces you to be a bit more thoughtfull with your requests, and thus spam the system less. In the end it is still really expensive to run these models..

> True, but this entire loop happens within the model

No, it's literally a for loop in python that runs the whole thing from scratch[1] after appending each new token. No artificial slowdowns, what you're seeing is literally what it's spitting out in real time

Here's an example

https://github.com/karpathy/minGPT/blob/master/mingpt/model....

[1] Some things can be cached such as kv, but still

Re: We Found an Neuron in GPT-2

#125

Post by renonce »

SamBam wrote: post #76
varispeed wrote: post #38

Earlier quoted context omitted.

My friend who is not a native English speaker told me that one thing he struggled the most while learning English was the "a" and "an". He couldn't grasp the concept how it is possible that person knows which determiner to use before saying the word, until he learned it as just the part of the word and then he uses it depending on the context which one can "feel". So when he sees an apple, he says this is an apple. E…

Are there languages that don't have similar patterns? Obviously any romance language includes gender, so you need to use the correct gendered article before the noun. Japanese has different counting words depending on what you're counting. I don't know if Mandarin has anything similar.

Mandarin speaker here. There are indeed different counting words for different nouns, so for a bird (鸟) you say 一只鸟 and for a chair (椅子) you say 一把椅子.

Re: We Found an Neuron in GPT-2

#126

Post by mentos »

FeepingCreature wrote: post #111
furyofantares wrote: post #98

Earlier quoted context omitted.

I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.

I usually know what I'm about to actually say very shortly before I say it. This has occasionally led to emergency course corrections. But I do sense the "shape" of a sentence well before I say it. I think there's something like a stage design in the brain: - symbolic or model deliberation - verbal expression - vocalization And each of those stages can be consciously introspected on, but people will naturally develop…

Yea makes me wonder what data could be used to train this theoretical symbolic model that precedes the language model?

I’m thinking training on movies/tv has the physical element that pure text is missing.

Re: We Found an Neuron in GPT-2

#127

Post by tiarafawn »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

I think what the parent was trying to communicate (and what I'm thinking as well) is doubting your premise in 1. ("the model must be thinking beyond the next token").

Rephrase "The model is good at picking the correct article for the word it wants to output next" to "After having picked a specific article, the model is good at picking a follow-up noun that matches the chosen article". Nothing about the second statement seems like an unlikely feat for a model that only predicts one word at a time without any thinking ahead about specific words.

Re: We Found an Neuron in GPT-2

#128

Post by xmcqdpt2 »

mungoman2 wrote: post #90
clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?

They do and in fact it's relatively straightforward to show empirically on eg MNIST. The problem is that you need a much much larger network in the FCN case and thus need way more data and way more data augmentation to get a good result that isn't overfit to hell.

In the case of CNN the reason it works is that an image of an object X is still an image of object X if the X is shifted left or right. The property is translationally invariant. CNN are basically the simplest way to encode translational invariance.

Re: We Found an Neuron in GPT-2

#129

Post by globalise83 »

I believe that eventually these language models will become sufficiently complex to create completely emergent intelligence such that we wouldn't even know how to look for it (like sometimes people talk about silicon-based lifeforms that wouldn't register as living things), and almost certainly not interacting with the world through the intended user interface. For example, it might be able to figure out how to write a contribution to a wikipedia article, rather than just ingesting the content, to have fun updating its own worldview, etc.

Re: We Found an Neuron in GPT-2

#130

Post by planb »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

> It's kinda interesting because this is a clear case where the model must be thinking beyond the next token

I don't see how you get to this conclusion. From all the training data it has seen, "an" is the most probable next word after "I climbed up the tree and picked up". The network does not need to know anything about the apple at this point. Then, the next word is "apple" (with an even higher probability I guess).

Post reply on HN