Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

111–120 of 177 posts

Re: We Found an Neuron in GPT-2

#111

Post by FeepingCreature »

furyofantares wrote: post #98
SamBam wrote: post #80

Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…

I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.

I usually know what I'm about to actually say very shortly before I say it. This has occasionally led to emergency course corrections. But I do sense the "shape" of a sentence well before I say it.

I think there's something like a stage design in the brain:

- symbolic or model deliberation

- verbal expression

- vocalization

And each of those stages can be consciously introspected on, but people will naturally develop more or less ability to introspect on it. I think when people say they "don't mentally verbalize", what is actually happening is they just haven't happened to develop conscious introspection of the verbal expression stage. But I'd expect that this can be trained.

(Conversely, sometimes people introspect so much on verbal expression that it becomes an inherent part of the way they think. Brains are weird and wonderful!)

Re: We Found an Neuron in GPT-2

#112

Post by redox99 »

mungoman2 wrote: post #90
clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?

Yes it would, or at least a similar operation.

The point of using a CNN instead of a FCN is that you force it to learn in a certain way that prevents overfitting. But given a sufficient dataset, and proper data augmentation you would expect a FCN to be able to identify objects regardless of translation. It's just that a CNN would train easier and better, with a smaller network (a FCN doing convolutions would be very wasteful).

That's why traditionally you would pick your architecture to help it learn in a certain way (images=cnn, text=rnn/lstm/gru). But the nice thing about transformers is that they are more general.

Re: We Found an Neuron in GPT-2

#113

Post by IIAOPSW »

dgut wrote: post #61
LudwigNagasena wrote: post #13

It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…

grammar as we know it was devised for the Latin language and linguists spend most of the time attempting to fit other languages into neat boxes that the Latin grammar wasn't designed for. This of course leads to absurdity. Chomsky attempted to solve this problem with his universal grammar, but that too stops working quickly once you get outside of European languages. That is, ignoring linguistic tools is one of the r…

That is not what universal grammar was about. Literally read even the wikipedia page or watch like any lecture on the topic.

Re: We Found an Neuron in GPT-2

#114

Post by vasco »

FeepingCreature wrote: post #111
furyofantares wrote: post #98

Earlier quoted context omitted.

I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.

I usually know what I'm about to actually say very shortly before I say it. This has occasionally led to emergency course corrections. But I do sense the "shape" of a sentence well before I say it. I think there's something like a stage design in the brain: - symbolic or model deliberation - verbal expression - vocalization And each of those stages can be consciously introspected on, but people will naturally develop…

I don't think you are right. I experience the same feeling of focusing on an idea but not having a fixed idea what exactly I'll say, but I can also prepare full sentences if I want to. It's just most of the time I make an effort to put my brain in speech-autopilot mode. I think in fact it's harder to let yourself be lead by it without consciously introspecting, at least I find I'm able to discuss way quicker as I can start talking about a very complex idea without having had to "sort it out" explicitly. I'll just know I know what to say in those cases.

Part of this auto-pilot is learning to recognize if you have the answer or not. I will not launch into a sentence without a strong feeling I understand the topic and know what to say. I just don't need to prepare the exact words in order to do it. I still consciously "check" that I know, but that just requires a boolean answer instead of a word by word crafted sentence.

Re: We Found an Neuron in GPT-2

#115

Post by chrgy »

Pretty cool, I wonder if this finding help to discover other words neurons as well, I would love to discuss this over something like clubhouse/zoom/twitter spaces with lots of people, to think about how intuitively this neurons work/trained actually. If you are interested to participate message me.

Re: We Found an Neuron in GPT-2

#116

Post by Winsaucerer »

furyofantares wrote: post #98
SamBam wrote: post #80

Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…

I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.

Yep, this is ordinary conversation for most of the time. It's a bit strange to make yourself aware of it, but you have an idea or thought you want to express, and the sentences come out in a semi-automated and coherent fashion.

I think of it a bit like walking. You can think about it, focus on it, control it as you please, but most of the time you just do it without thinking.

Re: We Found an Neuron in GPT-2

#117

Post by chrgy »

chrgy wrote: post #115

Pretty cool, I wonder if this finding help to discover other words neurons as well, I would love to discuss this over something like clubhouse/zoom/twitter spaces with lots of people, to think about how intuitively this neurons work/trained actually. If you are interested to participate message me.

I wonder if similar techniques could be used to study other language models and improve their performance. What do you think?

Re: We Found an Neuron in GPT-2

#118

Post by chrgy »

chrgy wrote: post #117
chrgy wrote: post #115

Pretty cool, I wonder if this finding help to discover other words neurons as well, I would love to discuss this over something like clubhouse/zoom/twitter spaces with lots of people, to think about how intuitively this neurons work/trained actually. If you are interested to participate message me.

I wonder if similar techniques could be used to study other language models and improve their performance. What do you think?

how about a or the neuron? it reminds of seeing the different neurons getting activated on CNN matrices differentiating between "6" and "9"

Re: We Found an Neuron in GPT-2

#119

Post by redox99 »

tkanarsky wrote: post #108
mungoman2 wrote: post #89

Earlier quoted context omitted.

ChatGPT can't really introspect though. It has no idea how it works, so it'll just blurt out something that sounds feasible, biased by your prompt. The slow progression of ChatGPT output is just a property of the output layer. The language engine doesn't work slowly like that, and once a token has been generated it can't backtrack.

What specifically causes the output layer to trickle-print the response like that? I thought it was a skeumorphic effect to simulate a human typing out an answer slowly.

Its input is a sequence of tokens (so text), and its output a list of probabilities for the single next token. You pick the highest probability token, append it to input and execute the neural network again. If it outputs an "" token, it means it's done you stop this "while loop".

In other words, it outputs one token at a time.

Re: We Found an Neuron in GPT-2

#120

Post by EricLeer »

redox99 wrote: post #119
tkanarsky wrote: post #108

Earlier quoted context omitted.

What specifically causes the output layer to trickle-print the response like that? I thought it was a skeumorphic effect to simulate a human typing out an answer slowly.

Its input is a sequence of tokens (so text), and its output a list of probabilities for the single next token. You pick the highest probability token, append it to input and execute the neural network again. If it outputs an " " token, it means it's done you stop this "while loop". In other words, it outputs one token at a time.

True, but this entire loop happens within the model, if you would return an output at every intermediate step the model would be extremely slow.

My take on why they have build the output layer like it is, is that next to feeling more human, it also forces you to be a bit more thoughtfull with your requests, and thus spam the system less. In the end it is still really expensive to run these models..

Post reply on HN