Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

131–140 of 177 posts

Re: We Found an Neuron in GPT-2

#131

Post by simonh »

ly3xqhl8g9 wrote: post #109
clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

Could a "type system" for neural weights be developed? Given a self-driving system, to be able to statically check that the neurons have the "Person" type, the "Don't Run Over Person" type, and so forth. What happens if you "transplant" the weights for ' an' to another network, some kind of transfer learning but componentized, does it still predict as accurately? If neural networks could be assembled from "types" it…

The way an LLM decides which word to use next is by evaluating the weightings of all the preceding words with every candidate word to calculate a probability for each of them. So if it selects ‘an’ as the next word, it’s because the weighting connecting ‘an’ to all the preceding words, and their orders in the text and relationships with each other predicted it should have a high probability of occurring.

So you can’t extract the weightings for ‘an’ discretely because those weightings encode its connection with all the other words and combinations and sequences or clusters of words it might ever be used with, including their weightings with other preceding words, and their relationships, etc, etc.

Re: We Found an Neuron in GPT-2

#132

Post by candiodari »

xmcqdpt2 wrote: post #128
mungoman2 wrote: post #90

Earlier quoted context omitted.

Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?

They do and in fact it's relatively straightforward to show empirically on eg MNIST. The problem is that you need a much much larger network in the FCN case and thus need way more data and way more data augmentation to get a good result that isn't overfit to hell. In the case of CNN the reason it works is that an image of an object X is still an image of object X if the X is shifted left or right. The property is tra…

> CNN are basically the simplest way to encode translational invariance

That's the geometric deep learning theory, isn't it? Do you know if there's a list somewhere of exactly what invariance has which ways to simulate it? Like an overview?

Re: We Found an Neuron in GPT-2

#133

Post by vasco »

FeepingCreature wrote: post #121
vasco wrote: post #114

Earlier quoted context omitted.

I don't think you are right. I experience the same feeling of focusing on an idea but not having a fixed idea what exactly I'll say, but I can also prepare full sentences if I want to. It's just most of the time I make an effort to put my brain in speech-autopilot mode. I think in fact it's harder to let yourself be lead by it without consciously introspecting, at least I find I'm able to discuss way quicker as I can…

I don't think it's like ... if an idea gets moved to the verbalization stage, that you're compelled to say it. It's more that some people seem to have little control over what happens at that stage, ie. they can operate on a concept but they can't predict the way it'll be spoken. Like [Conceptual stage] --??-- consciousness V [Verbalization stage] --??-- consciousness V [Vocalization] So if you don't have conscious a…

I understood your original argument, maybe I just did a bad job describing what I disagree with. In this post it's this:

> Conversely, if you don't have conscious access to conceptualization, you end up thinking that "thinking" always involves "thinking out loud", because "thinking out loud" (verbalizing) is the only way you have to query your conceptual layer. You literally only become aware of your own thinking after the thought is already pretty far along

What I'm saying is that even if you do have conscious access to verbalisation stage, the more you force yourself to let it be handled subconsciously, the better at communicating quickly and effectively. This is what I meant by autopilot.

Spending active conversation time on conscious verbalization seems to me as inefficient as verbalising words and "speaking them out" as you read a book, usually known as subvocalization.

Re: We Found an Neuron in GPT-2

#134

Post by moffkalast »

QuadmasterXLII wrote: post #68
Camillo wrote: post #66

Earlier quoted context omitted.

The grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.

I'd call it a pun

I'd call it an pun

Re: We Found an Neuron in GPT-2

#135

Post by pera »

josephmiller wrote: post #67
tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

I don't understand why your conclusion is that "the model must be thinking beyond the next token": the model doesn't need to do that to generate a well-formed sentence because it's not constrained by the size of the sentence.

Re: We Found an Neuron in GPT-2

#136

Post by naasking »

charcircuit wrote: post #87
nmfisher wrote: post #70

Earlier quoted context omitted.

I'm designing a Chinese learning app and I'd mostly agree with this. My working hypothesis though? Most adult learners don't have the time/patience for massive/lengthy exposure, so grammar lessons are a shortcut to "feeling" like they're making progress. I think it's a mistake to discount the psychology of learning. It's like saying that "calories-in, calories-out" is all there is to weight-loss. Strictly true, but n…

Grammar lessons are a shortcut. So are books and other corpuses of knowledge. People have spent time documenting patterns that exist and its useful to learn from them instead having to brute force everything yourself.

This is assuming that the brain regions that learn rules overlap with the brain regions that develops fluency in a language. I think Krashen's hypothesis is that this is largely not the case. You can "fake" some degree of competence by learning the rules and using that brain region, but you're slow and not fluent until you expose the other region to enough real-world data.

Re: We Found an Neuron in GPT-2

#137

Post by [deleted] »

clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

[deleted]

Re: We Found an Neuron in GPT-2

#138

Post by bnegreve »

Activations of individual neurons are hardly relevant, quoting Szegedy et al. from [1]

we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.

[1] Intriguing properties of neural networks

Re: We Found an Neuron in GPT-2

#139

Post by VeninVidiaVicii »

tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Word shmord, “an apple” is a lexical concept.

Re: We Found an Neuron in GPT-2

#140

Post by Rioghasarig »

Camillo wrote: post #66
hislaziness wrote: post #62

It is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.

The grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.

I think it was deliberate but just as a joke.
Post reply on HN