Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

61–70 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#61

Post by meepmorp »

zvmaz wrote: post #29

From the conclusion: > First, the fact that language models can be trained on large amounts of text data and can generate human-like language without any explicit instruction on gram- mar or syntax suggests that language may not be as biologically determined as Chomsky has claimed. Instead, it suggests that language may be learned and developed through exposure to language and interactions with others. I'm not a ling…

> For example, how is the fact that language models "work" contradict the innateness of language in humans?

It doesn't. Also, the author doesn't seem to actually understand Chomsky's writing about language, because learning language via exposure is how humans learn languages and he fucking mentions that in his writing on the subject.

UG (universal grammar) is the purported facility in the human brain which makes language possible - it has a innate structure, but it learns particular languages from exposure. Chomsky doesn't state exactly what that structure is because he doesn't know - figuring that out is goal of his work.

Re: Modern language models refute Chomsky’s approach to language

#62

Post by famouswaffles »

lisasays wrote: post #23
famouswaffles wrote: post #13

Earlier quoted context omitted.

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universa…

1. Measured how?

The closest biological analogue for an artificial parameter is a synapse. Even then, Synapses, and the brain in general, have more degrees of freedom than just a single floating point parameter allows.

Humans have some 50 to 100 trillion synapses. Who knows how far scaling goes in a transformer but simply increasing the parameter count increases performance so far.

Either way, we're not really close to emulating the complexity of the brain especially when taking neurons into account (one human neuron is far more complex than any one artificial parameter)

In other words, simpler building blocks and smaller sie. Of course, how much complexity is needed for Intelligence is unknown.

Re: Modern language models refute Chomsky’s approach to language

#63

Post by justrealist »

chubot wrote: post #52
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

There's really no good data to support this claim.

Re: Modern language models refute Chomsky’s approach to language

#64

Post by dr-detroit »

chubot wrote: post #52
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

[dead]

Re: Modern language models refute Chomsky’s approach to language

#65

Post by akprasad »

chubot wrote: post #52
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

The best example I'm aware of is Nicaraguan Sign Language [1].

> Steven Pinker, author of The Language Instinct, claims that "The Nicaraguan case is absolutely unique in history ... We've been able to see how it is that children—not adults—generate language, and we have been able to record it happening in great scientific detail. And it's the only time that we've actually seen a language being created out of thin air."

[1] https://en.wikipedia.org/wiki/Nicaraguan_Sign_Language

Re: Modern language models refute Chomsky’s approach to language

#66

Post by nhatcher »

renewiltord wrote: post #7

I only skimmed a few pages, but it doesn't appear to be about the Chomsky hierarchy - which was what I was curious about.

Chomsky did quite a lot of things in linguistics. This is about whether or not there is a universal grammar: https://en.wikipedia.org/wiki/Universal_grammar

There is a fairly interesting read from Steven Pinker: "The language instinct" about this topic.

Re: Modern language models refute Chomsky’s approach to language

#69

Post by williamcotton »

neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

What about the fact that trying to use Chomsky's approach to universal grammar does not result in working language models yet a statistical approach does?

The Norvig-Chomsky debate is kind of old at this point:

https://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fig...

Re: Modern language models refute Chomsky’s approach to language

#70

Post by [deleted] »

mjburgess wrote: post #41
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans learn from the structure of the world -- not the structure of language. LLMs cheat at generating text because they do so via a model of the statistical structure of text. We're in the world , it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily . What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all lik…

[deleted]
Post reply on HN