Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We Found an Neuron in GPT-2

clementneo.com

Post reply on HN
Print view View original

61–70 of 177 posts

Re: We Found an Neuron in GPT-2

#61

Post by dgut »

LudwigNagasena wrote: post #13

It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…

grammar as we know it was devised for the Latin language and linguists spend most of the time attempting to fit other languages into neat boxes that the Latin grammar wasn't designed for. This of course leads to absurdity. Chomsky attempted to solve this problem with his universal grammar, but that too stops working quickly once you get outside of European languages. That is, ignoring linguistic tools is one of the reasons GPT is successful.

Re: We Found an Neuron in GPT-2

#62

Post by hislaziness »

It is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.

Re: We Found an Neuron in GPT-2

#63

Post by IgorPartola »

clementneo wrote: post #53
kelnos wrote: post #21

Earlier quoted context omitted.

Yeah, my feeling here was that it's sort of tautological: if GPT predicts "a", then it must then predict a word that would follow "a" and not require "an" (and vice versa). And if you think about it from the opposite direction: if it's working out a response that is eventually going to have "apple" in it, then all the data it's trained on is going to cause it to predict "an" even before it needs to predict "apple". (…

The main issue is that GPT is fundamentally an autoregressive language model — it's only predicting the next token based on the prompt at a single time. Every time it wants to predict the next word, it adds the previously predicted word into the prompt, repeating the cycle. We can intuitively guess that the model is 'working out a response that is eventually going to have "apple" in it', but we don't actually know ho…

Do an improv exercise with a friend. Construct a story about a sentient apple one word at a time. I guarantee you that at one point one of you will set up the other with an “an” in your story. This is how I believe this works.

Re: We Found an Neuron in GPT-2

#64

Post by chimpansteve »

caf wrote: post #40
varispeed wrote: post #38

Earlier quoted context omitted.

My friend who is not a native English speaker told me that one thing he struggled the most while learning English was the "a" and "an". He couldn't grasp the concept how it is possible that person knows which determiner to use before saying the word, until he learned it as just the part of the word and then he uses it depending on the context which one can "feel". So when he sees an apple, he says this is an apple. E…

And of course it is not unheard of for a native English speaker to correct themselves if they change their mind about which noun to use. You can imagine being asked to very rapidly verbally classify fruits ("it's an apple, it's an apple, it's a pear, it's an apple,..") that you might well find yourself stumbling like "it's a.. an apple".

It's also not unheard of for native English speakers to use it differently (in speech, at least) as a consequence of regional accents. I would use "a history book", because my accent pronounces the h sound. My wife says "an istory book", because her accent drops the h.

But yeah, it should be easier when you're just dealing with text I guess

Re: We Found an Neuron in GPT-2

#65

Post by anacrolix »

rcoveson wrote: post #25
jakear wrote: post #4

Earlier quoted context omitted.

More accurately, it's whether the word that comes after starts with a vowel sound . This is why `an 'istoric` is correct, and `a historic` is correct, but `an historic` is incorrect (as famously used by Steven Colbert).

I'm always thrown when people write "an SQL parser". I'll bet roughly half this crowd is the opposite and cringe when they read "a SQL parser".

I'm a grammar Nazi but I can report neither gets my hackles up in this case!

Re: We Found an Neuron in GPT-2

#66

Post by Camillo »

hislaziness wrote: post #62

It is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.

The grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.

Re: We Found an Neuron in GPT-2

#67

Post by josephmiller »

tqi wrote: post #10

> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…

Author here! I think this is reasonable but I have two responses.

1. It's kinda interesting because this is a clear case where the model must be thinking beyond the next token, whereas in most contexts it's hard to say whether the model thinks ahead at all (although I would guess that it does most of the time).

2. More importantly, the key question here is how it works. We're not surprised that it has this behavior, but we want to understand which exact weights and biases in the network are responsible.

Note also that this is just the introductory sentence and the rest of the article would read exactly the same without it.

> it doesn't seem that surprising that there might be cases where there is a dominant bigram [...] that would trigger an "an" prediction, without actually predicting the following word first

btw I don't really understand what you mean by this. Bigrams can explain the second prediction in a two word pair but not the first.

Re: We Found an Neuron in GPT-2

#68

Post by QuadmasterXLII »

Camillo wrote: post #66
hislaziness wrote: post #62

It is interesting, when I (definitely not a bot) read the headline I thought the grammar was wrong. took me a while to that "an" was not an indefinite article here. In the article headline the first alphabets of each word is capitalised and somehow it was easier for me to understand what the "An" meant here.

The grammar _is_ wrong. It should have been "We found the 'an' neuron in GPT-2". Given the article's contents, it's hard to believe that the authors would make such a mistake; it was probably done deliberately, as clickbait.

I'd call it a pun

Re: We Found an Neuron in GPT-2

#69

Post by josephmiller »

nl wrote: post #9
unshavedyak wrote: post #6

I wonder if this stuff will ever be applicable to a person and a laptop (or if it is now?). Ie this seems like such a cool area to be in but the data volumes required are huge, complex, etc. Code is simple, cheap, lean, etc by comparison. Do we have any insight on how this area of research could be usable with less hardware and data? Is there a visible future where a guy and a laptop can make a big program? (without…

You could run this analysis on a laptop very easily. The pile-10k dataset they used for analysis is 33MB, and GPT2 runs ok on a CPU. For the full 10K analysis it's probably quicker to get a GPU though.

Author here!

This is correct. We did almost all this work on a Macbook Pro. Although for the pile-10k dataset analysis we used an A100 GPU because it would take many hours to run the whole thing through GPT-2 on a laptop.

Re: We Found an Neuron in GPT-2

#70

Post by nmfisher »

letmevoteplease wrote: post #60
Baeocystin wrote: post #58

Earlier quoted context omitted.

Approaches such as you describe have been the dominant method for decades. That we finally 'cracked' natural language generation with tools that literally encode nothing about grammar ahead of time is one hell of a lesson, early days as it is in the learning of it.

Reminds me of Stephen Krashen's input hypothesis of second-language acquisition. Krashen argues that consciously studying grammar is more or less useless, and only massive exposure to the language results in acquisition.[1] This is true in my experience. [1] https://en.wikipedia.org/wiki/Input_hypothesis

I'm designing a Chinese learning app and I'd mostly agree with this. My working hypothesis though? Most adult learners don't have the time/patience for massive/lengthy exposure, so grammar lessons are a shortcut to "feeling" like they're making progress.

I think it's a mistake to discount the psychology of learning. It's like saying that "calories-in, calories-out" is all there is to weight-loss. Strictly true, but not helpful for 90% of people.

Post reply on HN