81–90 of 177 posts
Earlier quoted context omitted.
I is a vowel so your rule about sound doesn’t apply here. It’s also not a rule I’ve ever heard, in school we’re taught that only vowels get a before them
https://www.ecenglish.com/learnenglish/lessons/vowel-sounds-... "Vowel Sounds"
Earlier quoted context omitted.
I think the lack of explicit linguistic tools is the key to success, forcing/enabling the generic model to learn implicit linguistic tools (there's some research identifying that analysis of specific linguistic phenomena happens at specific places in the NN layers) that work better than what we could implement. "It would be interesting to know how different a model would be if it operated on eg dependency trees inste…
Hmm I think it's sort of a hybrid. First off, we know that overall the concept of language is humans is an emergent phenomenon. It developed from natural selection from simple components so there's validity that the same thing can occur in an LLM where some overarching emergent structure develops from simple primitives. At the same time we do know that a sort of universal grammar exists among humans. Our language cap…
It could have been hallucinating, of course. But it does seem like it occasionally alters already generated words as it goes, at least I think I've seen it do that.
Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…
It’s notable how successful LLMs despite the lack of any linguistic tools in their architectures. It would be interesting to know how different a model would be if it operated on eg dependency trees instead of the linear list of tokens. Surely, the question of “a/an” would be solved with ease as the model would be required to come up with a noun token before choosing its determiner. I wonder if the developers of LLMs…
grammar as we know it was devised for the Latin language and linguists spend most of the time attempting to fit other languages into neat boxes that the Latin grammar wasn't designed for. This of course leads to absurdity. Chomsky attempted to solve this problem with his universal grammar, but that too stops working quickly once you get outside of European languages. That is, ignoring linguistic tools is one of the r…
I don’t think this is entirely fair. Generative grammars have been produced for a huge variety of non-European languages, even non-Indo-European languages, and can account for tremendous diversity in linguistic rules. Even languages without fixed word orders or highly synthetic languages can be represented.
Linguistics isn’t focused on the problem of outputting reasonable-sounding text responses. Instead, it seeks to transparently explain how language works and is structured, something that GPT does not do.
Earlier quoted context omitted.
Hmm I think it's sort of a hybrid. First off, we know that overall the concept of language is humans is an emergent phenomenon. It developed from natural selection from simple components so there's validity that the same thing can occur in an LLM where some overarching emergent structure develops from simple primitives. At the same time we do know that a sort of universal grammar exists among humans. Our language cap…
Lots of linguistics researchers would disagree about the existence and necessity of universal grammar or Chomsky's 'language acquisition device' and in fact the success of LLMs and statistical models which very clearly have no LAD and no universal grammar over feature engineering or grammar based schemes suggests the opposite.
Has it changed since then? Have they found a significant number of human civilizations that use divergent grammars?
Earlier quoted context omitted.
Reminds me of Stephen Krashen's input hypothesis of second-language acquisition. Krashen argues that consciously studying grammar is more or less useless, and only massive exposure to the language results in acquisition.[1] This is true in my experience. [1] https://en.wikipedia.org/wiki/Input_hypothesis
I'm designing a Chinese learning app and I'd mostly agree with this. My working hypothesis though? Most adult learners don't have the time/patience for massive/lengthy exposure, so grammar lessons are a shortcut to "feeling" like they're making progress. I think it's a mistake to discount the psychology of learning. It's like saying that "calories-in, calories-out" is all there is to weight-loss. Strictly true, but n…
Earlier quoted context omitted.
Do an improv exercise with a friend. Construct a story about a sentient apple one word at a time. I guarantee you that at one point one of you will set up the other with an “an” in your story. This is how I believe this works.
Your friend is thinking ahead that the next word would be apple When they say ‘an’. But GPT can’t think ahead what token it will add after the one it is on. Or can it? It could “predict” internally the word apple for the next “meaningful” word and output ‘an’ because of this.
That's interesting. I just asked ChatGPT to explain how it decides "a" vs "an" and it confirmed that it will retroactively change "a" to "an" when it finds that the following word sounds like it starts with a vowel sound. It could have been hallucinating, of course. But it does seem like it occasionally alters already generated words as it goes, at least I think I've seen it do that.
The slow progression of ChatGPT output is just a property of the output layer. The language engine doesn't work slowly like that, and once a token has been generated it can't backtrack.
Co-author here! I'm kind of surprised that this made it to the top of HN! This was a project in which Joseph and I tried to reverse engineer the mechanism in which GPT-2 predicts the word 'an'. It's crazy that large language models work so well just by being trained as a next-word-prediction model over a large amount of text data. We know how image models learn extract the features of an image through convolution[1],…