Could a NN be developed to help find what neurons do what? Or that could tell you itself which neurons do what?
We Found an Neuron in GPT-2
151–160 of 177 posts
Re: We Found an Neuron in GPT-2
#152Earlier quoted context omitted.
The way an LLM decides which word to use next is by evaluating the weightings of all the preceding words with every candidate word to calculate a probability for each of them. So if it selects ‘an’ as the next word, it’s because the weighting connecting ‘an’ to all the preceding words, and their orders in the text and relationships with each other predicted it should have a high probability of occurring. So you can’t…
Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" [1], and further more, group equivariant deep learning [2], maybe there is a way in which you can isolate a certain concept/"type", such as Person , Car , and so forth; perhaps not even isolate, but rehydrate the context of where the concept takes place: as a brain does in various word plays, as in Who's on First [3], etc. Com…
> Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron"
Firstly even if there is such a cell that only fires for one face, or perhaps also the person’s name, it doesn’t mean there aren’t other cells that fire for that person, or for people in general including that person. Without those as well, that neurons responses might not mean anything to the rest if the brain. It’s a thought experiment but never really demonstrated.
Also even if this is true in the very strongest sense. Say there is one neuron that uniquely and discretely fires in response to thinking about that one person. What defines a neuron isn’t just its internal behaviour. It’s also the pattern of inputs that influence it, and the pattern of outputs it sends out. It’s the connections and dependencies on the weightings and signals and responses from all the cells it’s connected to. Including the specific unique ways all those neurons are connected, or not connected to all the other cells in the brain. It’s al, the specifics of that connectedness that are what makes the behaviour of that neuron meaningful.
If you took that neuron and implanted it into another brain, you’d need to hook it up to the neurons in that brain such that it gets exactly the same stimuli, in the same order, with the same strength, every time it needs to fire. The same applies to its output, all the neurons it’s connected to would have to interpret its firing behaviour in the exact same way the other neurons in the original brain did. But there’s no guarantee any of those connected mechanisms work or are physically connected in the same way, or even a vaguely similar or compatible way in the new brain.
Re: We Found an Neuron in GPT-2
#153Earlier quoted context omitted.
Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" [1], and further more, group equivariant deep learning [2], maybe there is a way in which you can isolate a certain concept/"type", such as Person , Car , and so forth; perhaps not even isolate, but rehydrate the context of where the concept takes place: as a brain does in various word plays, as in Who's on First [3], etc. Com…
> Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" Firstly even if there is such a cell that only fires for one face, or perhaps also the person’s name, it doesn’t mean there aren’t other cells that fire for that person, or for people in general including that person. Without those as well, that neurons responses might not mean anything to the rest if the brain. It’s a thoug…
Well, given the more organic nature of machine learning and what it's trying to achieve I wouldn't be surprised if that same neuron also triggered to some degree for "Jennifer and Stefan" ahaha.
Re: We Found an Neuron in GPT-2
#154Earlier quoted context omitted.
So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? How is it "clocked" to get successive tokens?
> So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? Yeah except it doesn't necessarily take the strongest one, you can set a 'temperature' parameter which determines how often you should pick the second choice, etc. >How is it "clocked" to get successive tokens? You add each new token to the input prompt to generate the next one.
>> You add each new token to the input prompt to generate the next one.
OMFG so it really is a super advanced auto-complete!!
Yeah, this is not AGI or anywhere close. That explains how it picks up context, and also how it can lose the plot after a bit due to limited input size.
Re: We Found an Neuron in GPT-2
#155Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…
I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning). This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?). Here he asks it to write something using Kurt Vonnegut's rules of writing. https://twitter.com/emollick/status/1626084142239649792 It seems hard to explain how Bing/…
There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.
Re: We Found an Neuron in GPT-2
#156Earlier quoted context omitted.
I don't think it's like ... if an idea gets moved to the verbalization stage, that you're compelled to say it. It's more that some people seem to have little control over what happens at that stage, ie. they can operate on a concept but they can't predict the way it'll be spoken. Like [Conceptual stage] --??-- consciousness V [Verbalization stage] --??-- consciousness V [Vocalization] So if you don't have conscious a…
I understood your original argument, maybe I just did a bad job describing what I disagree with. In this post it's this: > Conversely, if you don't have conscious access to conceptualization, you end up thinking that "thinking" always involves "thinking out loud", because "thinking out loud" (verbalizing) is the only way you have to query your conceptual layer. You literally only become aware of your own thinking aft…
Oh yeah I agree with that. I think the conscious access is primarily important for self-training. So I would expect somebody with conscious access to 1. be better at vocalizing, 2. almost never actually need to use their access to correct a decision.
I tend to think of consciousness as the "debug mode" of the brain.
Re: We Found an Neuron in GPT-2
#157Earlier quoted context omitted.
I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning). This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?). Here he asks it to write something using Kurt Vonnegut's rules of writing. https://twitter.com/emollick/status/1626084142239649792 It seems hard to explain how Bing/…
There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.
Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").
Re: We Found an Neuron in GPT-2
#158Earlier quoted context omitted.
There is "long range" dependence, it's just only on the prompt: the conversation with the user and the hidden header (e.g. "Answer as ChatGPT, an intelligent AI, state your reasons, be succinct, etc."). That ends up being enough.
Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").
Its chosen by the ngram and randomly so, that does suggest it is completing the text a word at a time.
Re: We Found an Neuron in GPT-2
#159Earlier quoted context omitted.
Sure, but the point being discussed is that despite the word by word output, the output does not appear to be "chosen" on a word by word basis. OP investigated the case where the word "an" anticipates the following word ("an apple" vs "a pear").
Its chosen by the ngram and randomly so, that does suggest it is completing the text a word at a time.
Did you check the Vonnegut writing rules example I posted at top of this thread - in particular look at Bing/GPT's explanation of how its cake story matches up to Vonnegut's rules ? It's hard to imagine how it could have come up with such a coherent story, checking all the rules, if it was only conceiving of it's continuing story on a word by word basis. It's not as if sentence #1 matches rule number 1, sentence 2 matches rule number 2, etc. It seems there had to be some wholistic composition for it to do that.
Note too that despite the output being sampled from a distribution based on a "randomness" temperature, there are many case where what it is trying to say so much constrains the output that certain words/synonyms/concepts are all but forced.
Re: We Found an Neuron in GPT-2
#160Earlier quoted context omitted.
Several papers have explored emergent linguistic structure in LMs. Here's some early introductory work in this space. Despite having explicit syntax parses etc as input, models seem to learn something like syntax. https://arxiv.org/abs/1905.05950 https://aclanthology.org/N19-1419/ https://arxiv.org/abs/1906.04341
I would add https://www.pnas.org/doi/10.1073/pnas.1907367117
Yeah this is more or less a re-release of the second work I linked, except for a broader audience and more targeted at linguistics / cognitive science researchers.