Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

71–80 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#71

Post by OskarS »

chubot wrote: post #52
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

The classic example which I think you’re referring to is Nicaraguan Sign Language, which developed organically in Nicaraguan schools for deaf children where neither the children nor the teachers knew any other form of sign language. It’s a fascinating story, a complex fully developed language created by children. Seems to indicate that this is indeed a very inate capability among humans in larger groups: https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language

Re: Modern language models refute Chomsky’s approach to language

#73

Post by dahart »

hwillis wrote: post #30
famouswaffles wrote: post #13

Earlier quoted context omitted.

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universa…

> No Language model is yet even close to the scale of the human brain If GPT-4 has 100 trillion parameters, it has as many parameters as the human brain has synapses. Synapses are a lot simpler than parameters; they're digital. A single neuron needs many synapses, all of roughly equal weight, emitting many pulses over a short time in order to convey a single weighted value. On top of that, you may have heard that the…

Google says gpt 4 has ~1 trillion parameters.

Re: Modern language models refute Chomsky’s approach to language

#74

Post by somewhereoutth »

mjburgess wrote: post #41
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans learn from the structure of the world -- not the structure of language. LLMs cheat at generating text because they do so via a model of the statistical structure of text. We're in the world , it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily . What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all lik…

Exactly. A very uncomfortable truth for those heavily invested (time/money/credence) in this latest AI wave.

Re: Modern language models refute Chomsky’s approach to language

#75

Post by [deleted] »

CydeWeys wrote: post #12
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

[deleted]

Re: Modern language models refute Chomsky’s approach to language

#76

Post by aap_ »

zvmaz wrote: post #42
aap_ wrote: post #27

It sure is an interesting time for linguistics now. For the first time there now is a second entity that can understand and produce language. Will be exciting to see how the field will transform.

> For the first time there now is a second entity that can understand and produce language But does it really understand language? This reminds me of the Chinese room argument [1]. [1] https://en.wikipedia.org/wiki/Chinese_room

Of course this getting into the whole philosophy of AI and consciousness and whatnot. But for practical purposes I would say: yes

Re: Modern language models refute Chomsky’s approach to language

#77

Post by CydeWeys »

famouswaffles wrote: post #18
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans invented English, and every other human language. Not one single human, and not in a single lifetime, but yes, we did it.

LLMs are not doing this.

Re: Modern language models refute Chomsky’s approach to language

#78

Post by civilitty »

lisasays wrote: post #23
famouswaffles wrote: post #13

Earlier quoted context omitted.

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universa…

1. Measured how?

The connectome [1]. There are about 100B +- 20B neurons in the average human brain. Each neuron has thousands of synapses and each synapse is made up of dozens to hundreds of neurotransmitter channels.

That’s at least a quadrillion parameters which is at least three orders of magnitude bigger than SOTA LLMs assuming each of those synapse channels maps to a parameter. That is an absurd assumption given neuroplasticity, which operates at a high level to adapt the neural network as it learns. See the plasticity section in the linked wikipedia articles: connections between neurons can grow or even get removed on the scale of hours and days. The human brain has entire biological systems supporting intelligence for which there is no real ML equivalent because our ML architectures are static.

Given a family member’s research on neurotransmitter potentiation, I’d estimate 10 to 100 parameters per channel to get full fidelity of the human brain. (This is absurdly speculative - we have no idea at which fidelity will intelligence emerge)

[1] https://en.m.wikipedia.org/wiki/Connectome

Re: Modern language models refute Chomsky’s approach to language

#79

Post by michaelhartm »

Nobody knows how the LLMs work under the hood. It's just lots of stacked transformers that encode various concepts. Nothing in this book refutes whether Chomsky's concepts are actually being encoded in LLMs or not. For all we know, Chomsky's concept of "binding principles", "binary branching" etc could be represented inside the inner layers of these many billion parameter models. In fact, I'd argue that this is the right research to do. Prove that no transform or feed-forward layer inside the neural net encodes, say "binding principles".

Re: Modern language models refute Chomsky’s approach to language

#80

Post by mempko »

williamcotton wrote: post #69
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

What about the fact that trying to use Chomsky's approach to universal grammar does not result in working language models yet a statistical approach does? The Norvig-Chomsky debate is kind of old at this point: https://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fig...

Because Chomsky is trying to build a bird, and Norvig is trying to build an airplane. It's much easier to build an airplane to fly than a bird.

Chomsky is trying to explain how humans create language. LLM are creating language, but not the way humans do.

Nothing about this paper refutes Chomsky's claims.

Post reply on HN