We found a Neuron in a Neural Network Nothing new to see here. They pinpoint the nodes where the training bumped up the numbers for one token while not firing other tokens. Yes, I’m being a bit reductionist, but GPT -> transformer architecture -> neural network. We just have more detailed techniques, a lot more data, storage, and processing power now. But the basics of how a NN works hasn’t changed.
The title is a pun. The interest here is that this kind of highly-specific node activation analysis is still new in text models, unlike in image models where it's been thoroughly explored.
It goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not…
Further to this, as I understand it the "embedding" is mapping the tokens into vectors in a space where tokens semantically similar will be close together in the space.
Lest we forget that the ability to even construct this sort of "word vector" was somewhat recently considered both fascinating and cutting-edge (word2vec turns 10 this year). We have come a very long way, very quickly.
It goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not…
Surely "an eagle" is [ an][ eagle] since the an starts a new word.
The OpenAI tokenizer tool agrees with you, unless it's at the start of the document. e.g. "An eagle flew into its nest ..."
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
When you say "they have no idea about what they're about to say" you're talking about conscious thought. I think there is a difference between rational thought (thinking by going through a series of logically connected steps) and intuition, where you can arrive at a conclusion or knowledge of some fact or concept or knowing how to do something, without having gone through those conscious steps. Does one count as "thoughts" any less than the other? People are sometimes so quick to dismiss any subconscious thinking as being nothing more than a very complicated computer, but I couldn't disagree more.
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
> "the discussion is an... an... epistemological one"
I hear that in the voice of Agent Smith, from The Matrix.
Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…
Convolution is part of the network design though. Would a fully connected network learn to convolute? Or would it turn out that convolution is not necessary?
The interesting part here isn't the convolution itself, it's how convolutional layers turn out to like "filters" or "detectors" for individual features. This is explained very well in the distill.pub article linked by GP.
We know the architecture of LLMs because we created it, but we don't yet have the same level of understanding about them, or the same quality of analytical tools for reasoning about them.
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
When you say "they have no idea about what they're about to say" you're talking about conscious thought. I think there is a difference between rational thought (thinking by going through a series of logically connected steps) and intuition, where you can arrive at a conclusion or knowledge of some fact or concept or knowing how to do something, without having gone through those conscious steps. Does one count as "tho…
Well I think that's the point. We know that our minds engage in a large amount of pattern matching/recognition/retrieval. Perhaps this hugely-powerful pattern-matching information-retrieval engine has learned to perform similarly to human minds, being trained exclusively on the output of human minds.
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
I never know what I'm about to say, but somehow coherent sentences come out.
I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
When you say "they have no idea about what they're about to say" you're talking about conscious thought. I think there is a difference between rational thought (thinking by going through a series of logically connected steps) and intuition, where you can arrive at a conclusion or knowledge of some fact or concept or knowing how to do something, without having gone through those conscious steps. Does one count as "tho…
I tend to take the view that thoughts are very similar to sensory input. if you sit in silence for 60 seconds, you literally do not and can not predict what thoughts pop up in consciousness. You can actively focus attention on a thought, but if you try to find the source of the thought, it disapears. Thoughts just appear, just like sounds, sight, etc just appear.
Interestingly I feel like humans have this as well, sometimes. Sometimes if someone is working though a complex thought and they're not really sure where they're going, they'll pause while thinking of the word they want to use, and might sound like "the discussion is an... an... epistemological one" Obviously they may have been conscious that the next word was going to start with "epi..." and they are just trying to…
I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.
This is so weird. I am very “In my head” and have generally planned what I’m going to say before I open my mouth.
It makes me feel like I’m not very good at conversation though, especially smalltalk.