Modern language models refute Chomsky’s approach to language • Hacker News Board

Re: Modern language models refute Chomsky’s approach to language

#241

Post by mcv » Wed, Jun 14, 2023, 2:51 PM UTC

steveBK123 wrote: post #240

mcv wrote: post #156

Earlier quoted context omitted.

I think AGI is a questionable concept. We still don't have a good definition of what intelligence really is, and some people keep moving the goal posts. What we need is AI that fills specific needs we have.

If we simply AGI to be "general purpose AI", then my argument is - maybe the approach of LLMs works fine enough for textual generation, but it is not a path towards "general purpose AI".. and what we are going to have is different approaches for different niche use cases. I'm less convinced there's any unified solution for "general purpose AI" before us here.

And I'm convinced we don't even want "general purpose AI". We want AI for a variety of specific purposes. Admittedly these LLMs are a lot broader than I ever imagined, but they're still limited to generating text. I wouldn't want ChatGPT to drive my car.

Re: Modern language models refute Chomsky’s approach to language

#242

Post by dllthomas » Wed, Jun 14, 2023, 4:25 PM UTC

EGreg wrote: post #233

dllthomas wrote: post #227

Earlier quoted context omitted.

As I understand it, the question was whether humans learn language starting from a "blank slate" or whether there's some meaningful structure built in (that's therefore common across all human languages). Chomsky's argument was that children are not exposed to enough example sentences to learn all the rules they learn. It's an argument from the information content of the corpus made available to the language learner,…

That makes no sense. A “vastly larger corpus” would not have the property of “not being exposed to enough example sentences”. If anything, the amount of sentences is far far more than any human would ever encounter — let alone a child.

Right, that's the point - success with a huge corpus doesn't tell us it would have been possible with a realistic corpus, so doesn't refute the argument.

Re: Modern language models refute Chomsky’s approach to language

#243

Post by paulmd » Wed, Jun 14, 2023, 5:11 PM UTC

mjburgess wrote: post #172

dTal wrote: post #113

Earlier quoted context omitted.

There's nothing qualitatively less "in the world" about a language model than a human. Yes, a human has more senses, and is doubtless exposed to huge categories of training data that a language model doesn't have access to - but it's false to draw a sharp dichotomy between knowing what an iPhone looks like, and knowing how people talk about iPhones. Consider two people - one, a Papau New Guinea tribesperson from a pr…

LLMs don't have any senses, not merely fewer. LLMs don't have any concepts, not merely named ones. A concept is a sensory-motor technique abstracted into a pattern of thought developed by an animal, in a spatio-temporal environment, for a purpose. LLMs are just literally an ensemble of statistical distributions over text symbols. In generating text, they're just sampling from a compressed bank of all text ever digiti…

> A concept is a sensory-motor technique abstracted into a pattern of thought developed by an animal, in a spatio-temporal environment, for a purpose.

Hi, since human linguistics is the sole repository of linguistic conceptualism, can you please show me which of the neurons is the "doggie" neuron, or the "doggie" cluster of neurons? I want to know which part of the brain represents the thing that goes wag-wag.

If you can't mechanically identify the exact locality of the mechanism within the system, it doesn't really exist, right? It's just a stochastic, probabilistic model, humans don't understand the wag-wag concept, they just have some neurons that are weighted to fire when other neurons give them certain input stimuli tokens, right?

This is the fundamental problem: you are conflating the glue language with the implementation language in humans too. Human concepts are a glue-language thing, it's an emergent property of the C-language structure of the neurons. But there is no "doggie" neuron in a human just like there is no "doggie" neuron in a neural net. We are just stochastic machines too, if you look at the C-lang level and not the glue-language level.

Re: Modern language models refute Chomsky’s approach to language

#244

Post by mLuby » Fri, Jun 16, 2023, 3:53 PM UTC

AbrahamParangi wrote: post #38

steveBK123 wrote: post #17

Earlier quoted context omitted.

doesn't this make LLM a dead end towards AGI and mostly just a neat specific trick?

In order to believe this, you'd need to be able to imagine a specific test of something that an LLM could not do under any circumstances. Previously, that test could have been something like "compose a novel sonnet on a topic". Today, it is much less clear that such a test (that won't be rapidly beaten) even exists.

How about make new scientific discoveries?