so evidence for a grandmother cell (https://en.wikipedia.org/wiki/Grandmother_cell)
We Found an Neuron in GPT-2
11–20 of 177 posts
Re: We Found an Neuron in GPT-2
#12> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…
Not to mention that the corpus mostly will have the correct case for most common words like apple. Train it with essays of ESL students and you'll get something else.
Re: We Found an Neuron in GPT-2
#13It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs explored those approaches but found them infeasible due to large preprocessing times, immaturity of such tools and/or little benefit.
Re: We Found an Neuron in GPT-2
#14I wonder if this stuff will ever be applicable to a person and a laptop (or if it is now?). Ie this seems like such a cool area to be in but the data volumes required are huge, complex, etc. Code is simple, cheap, lean, etc by comparison. Do we have any insight on how this area of research could be usable with less hardware and data? Is there a visible future where a guy and a laptop can make a big program? (without…
Does this hypothetical laptop have a GPU? StableDiffusion is in this realm of "stuff" and is runnable on consumer GPU systems. It's a bit of trouble to get setup if you're not a python dev (and kinda still is if you are) but it's a pretty neat ML model to play around with.
Re: We Found an Neuron in GPT-2
#15It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…
Not relevant but:
"Every time I fire a linguist, the performance of the speech recognizer goes up". - Frederick Jelinek
Re: We Found an Neuron in GPT-2
#16> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…
Wouldn’t this also be correlated somehow in its vector space?
Re: We Found an Neuron in GPT-2
#17It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…
Not relevant but: "Every time I fire a linguist, the performance of the speech recognizer goes up". - Frederick Jelinek
Surely the remaining linguists muster up some improvements out of fear for their jobs!
Re: We Found an Neuron in GPT-2
#18> We started out with the question: How does GPT-2 know when to use the word an over a? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 is only capable of predicting one word at a time. We still don’t have a full answer... I'm not sure I understand why this is an open question. While I get that GPT-2 is predicting only one word at a time, it doesn't seem that surprising t…
From my naïve point of view it seems obvious that at any point where both an „a“ or a „an“ would fit the model randomly selects one of them and by doing so reduces the set of possible nouns to follow.
Re: We Found an Neuron in GPT-2
#19> The choice depends on whether the word that comes after starts with a vowel or not Or an aitch / h, but only sometimes.
More accurately, it's whether the word that comes after starts with a vowel sound . This is why `an 'istoric` is correct, and `a historic` is correct, but `an historic` is incorrect (as famously used by Steven Colbert).
I is a vowel so your rule about sound doesn’t apply here. It’s also not a rule I’ve ever heard, in school we’re taught that only vowels get a before them
Re: We Found an Neuron in GPT-2
#20Earlier quoted context omitted.
More accurately, it's whether the word that comes after starts with a vowel sound . This is why `an 'istoric` is correct, and `a historic` is correct, but `an historic` is incorrect (as famously used by Steven Colbert).
I is a vowel so your rule about sound doesn’t apply here. It’s also not a rule I’ve ever heard, in school we’re taught that only vowels get a before them
[dead]