Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

141–150 of 177 posts

Re: We Found an Neuron in GPT-2

#141

Post by ly3xqhl8g9 »

simonh wrote: post #131
ly3xqhl8g9 wrote: post #109

Earlier quoted context omitted.

Could a "type system" for neural weights be developed? Given a self-driving system, to be able to statically check that the neurons have the "Person" type, the "Don't Run Over Person" type, and so forth. What happens if you "transplant" the weights for ' an' to another network, some kind of transfer learning but componentized, does it still predict as accurately? If neural networks could be assembled from "types" it…

The way an LLM decides which word to use next is by evaluating the weightings of all the preceding words with every candidate word to calculate a probability for each of them. So if it selects ‘an’ as the next word, it’s because the weighting connecting ‘an’ to all the preceding words, and their orders in the text and relationships with each other predicted it should have a high probability of occurring. So you can’t…

Right, but if there is such a thing as the very plastically named "Jeniffer Aniston neuron" [1], and further more, group equivariant deep learning [2], maybe there is a way in which you can isolate a certain concept/"type", such as Person, Car, and so forth; perhaps not even isolate, but rehydrate the context of where the concept takes place: as a brain does in various word plays, as in Who's on First [3], etc.

Come to think of it, when someone teaches me a new concept, the principle of mass conservation, for instance, in some sense they are transferring their embedding into my brain, further on I will relate to mass conservation through what that person taught me. The transfer is a very lossy process, sure, but a transfer with reintegration nonetheless. Perhaps "mortal computation" [4] is a requirement.

[1] https://en.wikipedia.org/wiki/Grandmother_cell

[2] https://www.youtube.com/playlist?list=PL8FnQMH2k7jzPrxqdYufo...

[3] https://www.youtube.com/watch?v=kTcRRaXV-fg

[4] Geoffrey Hinton, The Forward-Forward Algorithm: Some Preliminary Investigations, chapter 8, https://www.cs.toronto.edu/~hinton/FFA13.pdf

Re: We Found an Neuron in GPT-2

#142

Post by compressedgas »

KhoomeiK wrote: post #44
LudwigNagasena wrote: post #13

It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…

Several papers have explored emergent linguistic structure in LMs. Here's some early introductory work in this space. Despite having explicit syntax parses etc as input, models seem to learn something like syntax. https://arxiv.org/abs/1905.05950 https://aclanthology.org/N19-1419/ https://arxiv.org/abs/1906.04341

I would add https://www.pnas.org/doi/10.1073/pnas.1907367117

Re: We Found an Neuron in GPT-2

#143

Post by edulix »

Winsaucerer wrote: post #116
furyofantares wrote: post #98

Earlier quoted context omitted.

I never know what I'm about to say, but somehow coherent sentences come out. I certainly don't know what words a sentence is going to end with when I'm thinking or saying the first words in the sentence. I just think or say the sentence from start to finish, never knowing what the next word is going to be as I'm thinking the current one, and by the end of it I've thought or said a full sentence that makes sense.

Yep, this is ordinary conversation for most of the time. It's a bit strange to make yourself aware of it, but you have an idea or thought you want to express, and the sentences come out in a semi-automated and coherent fashion. I think of it a bit like walking. You can think about it, focus on it, control it as you please, but most of the time you just do it without thinking.

A way to think about it is a programmer: your mind is just a lot of functions - written by the `consciousness()` function, that has the main loop. You have the `moveLeftFootUp()` function, that can be called by the `walk(speed="normal")` function, that can be called by the `morningWalk()` function, etc.

Consciousness is the caller. You/It can consciously manually call `moveLeftFootUp()`, then `moveRightFootDown()`. Or maybe you were calling `walk(speed="normal")` and stepped in and started debugging that function's code at that level, step by step. Also, these functions sometimes raise exceptions, which are either handled by the function's caller automatically or bringing it to the caller's attention (i.e. the `conciousness()` main loop).

Learning to walk involves first manually calling `moveLeftFootUp()` and `moveRighFootDown()` order (once you have drafted those functions) in different order to get right how that should be done, then prototyping some `walk()` function code. The initial version of the `walk()` function at the begining isn't very robust and doesn't handle a lot of edge cases, thus raising exceptions all the time and requiring a lot of conscious effort. Of course, you are also adjusting `moveRighFootDown()` and `moveLeftFootUp()` at the same time or maybe creating `moveFoot(feet,direction)` function, etc.

But in the end, after all the fine adjustments of the code, you basically get the code for `walk()` right, it stops raising exception's to the main loop and doesn't require too much effort. You can just call the `walk()` function and it just works automatically (unless you step in with the debugger) - or you can continue creating new functions that call `walk()` inside those, confidently.

Re: We Found an Neuron in GPT-2

#144

Post by HarHarVeryFunny »

clementneo wrote: post #75

Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…

I wonder if you could comment on this (related to question of how far ahead these "LLM"s are planning).

This is Wharton professor Ethan Mollick playing with the new Bing chat, which seems considerably more advanced than ChatGPT (based on GPT-4 perhaps?).

Here he asks it to write something using Kurt Vonnegut's rules of writing.

https://twitter.com/emollick/status/1626084142239649792

It seems hard to explain how Bing/GPT could have generated the Vonnegut-inspired cake story, having ingested the rules, without planning the whole thing before generating the first word.

It seems there's an awful lot more going on internally in these models than a mere word by word autoregressive generation. It seems the prompt (in this case including Vonnegut's rules) is ingested and creates a complex internal state that is then responsible for the coherency and content of the output. The fact that it necessarily has to generate the output one word at a time seems to be a bit misleading in terms of understanding when the actual "output prediction" takes place.

Re: We Found an Neuron in GPT-2

#145

Post by sebzim4500 »

tiarafawn wrote: post #127
josephmiller wrote: post #67

Earlier quoted context omitted.

Author here! I think this is reasonable but I have two responses. 1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time). 2. More importantly, the key question here is how it works. We're not surprised that it has this behavior,…

I think what the parent was trying to communicate (and what I'm thinking as well) is doubting your premise in 1. ("the model must be thinking beyond the next token"). Rephrase "The model is good at picking the correct article for the word it wants to output next" to "After having picked a specific article, the model is good at picking a follow-up noun that matches the chosen article". Nothing about the second stateme…

The prompt is:

>I climbed up the pear tree and picked a pear. I climbed up the apple tree and picked

The argument made in the article (IMO an extremely convincing one) is that it wouldn't be able to predict the word 'an' except by observing that the word afterwards must be apple. Otherwise why not pick 'a'?

Re: We Found an Neuron in GPT-2

#146

Post by sebzim4500 »

Camillo wrote: post #66
hislaziness wrote: post #62

It is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.

The grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.

Not every attempt to make a humorous/engaging title is 'clickbait'.

Re: We Found an Neuron in GPT-2

#147

Post by sebzim4500 »

phkahler wrote: post #57
vore wrote: post #34

Earlier quoted context omitted.

It goes something like this: input text -> input tokens -> input embeddings -> model -> output embeddings -> output tokens -> output text Tokens aren't necessarily words: they can be fragments of words and you can check out this behavior here: https://platform.openai.com/tokenizer For instance, "an eagle" is tokenized to [an][ eagle], but "anoxic" is tokenized to [an][oxic], so just looking for the [an] token is not…

So is the output compared to the embedding vectors (via dot product) and the strongest one output its token? How is it "clocked" to get successive tokens?

> So is the output compared to the embedding vectors (via dot product) and the strongest one output its token?

Yeah except it doesn't necessarily take the strongest one, you can set a 'temperature' parameter which determines how often you should pick the second choice, etc.

>How is it "clocked" to get successive tokens?

You add each new token to the input prompt to generate the next one.

Re: We Found an Neuron in GPT-2

#148

Post by sebzim4500 »

johntb86 wrote: post #104

Is the " an" token the only way GPT-2 will ever produce the string " an"? Or will it sometimes combine separate " a" "n" tokens? I suppose separate tokens like that won't be seen in the input string, so they'll never be predicted?

Yeah, tokens exist for [ a] and [n] but during training it would still be considered incorrect to produce [ a][n] rather than [ an], so the model will never learn to output them.

Re: We Found an Neuron in GPT-2

#149

Post by sebzim4500 »

bfung wrote: post #73

We found a Neuron in a Neural Network Nothing new to see here. They pinpoint the nodes where the training bumped up the numbers for one token while not firing other tokens. Yes, I’m being a bit reductionist, but GPT -> transformer architecture -> neural network. We just have more detailed techniques, a lot more data, storage, and processing power now. But the basics of how a NN works hasn’t changed.

Are you really not surprised by this? Maybe I haven't been following the space, but I would have expected to see loads of neurons in the same layer which together determine whether [ an] or [ a] should be generated.

Not one neuron which was more relevant than all the others put together.

Post reply on HN