Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

41–50 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#41

Post by mjburgess »

famouswaffles wrote: post #18
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans learn from the structure of the world -- not the structure of language.

LLMs cheat at generating text because they do so via a model of the statistical structure of text.

We're in the world, it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily.

What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all like an iphone.

We have just stipulated this connection. The machine replays these stipulations to us -- it does not make them, as we do.

Re: Modern language models refute Chomsky’s approach to language

#42

Post by zvmaz »

aap_ wrote: post #27

It sure is an interesting time for linguistics now. For the first time there now is a second entity that can understand and produce language. Will be exciting to see how the field will transform.

> For the first time there now is a second entity that can understand and produce language

But does it really understand language? This reminds me of the Chinese room argument [1].

[1] https://en.wikipedia.org/wiki/Chinese_room

Re: Modern language models refute Chomsky’s approach to language

#43

Post by cma »

renewiltord wrote: post #7

I only skimmed a few pages, but it doesn't appear to be about the Chomsky hierarchy - which was what I was curious about.

Deep Mind had this paper on that, showing weaknesses of transformers relative to other models in generalizing from example productions from different tiers of the hierarchy:

https://arxiv.org/abs/2207.02098

Re: Modern language models refute Chomsky’s approach to language

#44

Post by jrflowers »

MrBuddyCasino wrote: post #15
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

Wordcels think LLMs imitate the human brain, when a shape rotator knows they really just imitate human language.

I love these new terms, can you elaborate on this?

Re: Modern language models refute Chomsky’s approach to language

#45

Post by guerrilla »

neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> I also respond to several critiques of large language models, including [...] skepticism that they are informative about real life acquisition

Yeah the whole thing hinges on this... and uh yeah good luck with that one...

Re: Modern language models refute Chomsky’s approach to language

#46

Post by taeric »

zvmaz wrote: post #29

From the conclusion: > First, the fact that language models can be trained on large amounts of text data and can generate human-like language without any explicit instruction on gram- mar or syntax suggests that language may not be as biologically determined as Chomsky has claimed. Instead, it suggests that language may be learned and developed through exposure to language and interactions with others. I'm not a ling…

It contradicts the idea that you have to teach language using syntax and grammar. Which... I confess I thought was already not believed? We certainly aren't teaching kids in the home how to decline and conjugate words. Are we?

(Similarly, languages that have gender are typically just picked up by usage, not necessarily ingrained by reasoning. Which leads to the obvious bad results when people think that there was solid reasoning on those choices, in the first place.)

Re: Modern language models refute Chomsky’s approach to language

#47

Post by kristopolous »

asdff wrote: post #33
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

Imagine being told all you need to do to learn Spanish, is to read a 300,000 word Spanish dictionary end to end so that you can probalistically come up with 1000 conversational phrases. Anyone who has learned a language can tell you it just doesn't work like that. You don't work by accumulating a massive dataset and training on it. No one can hold such a massive dataset of anything in their head at once.

Going from zero to one is different than one to two ("second language acquisition")

Zero to one is closer to mimicry and immersion. There's a long Wikipedia article on the field of study https://en.m.wikipedia.org/wiki/Language_acquisition

Furthermore, humans probably aren't static learners and likely have more beneficial times of certain study than others. There's a theory in that too https://en.m.wikipedia.org/wiki/Critical_period_hypothesis

Saying there's a "digital brain" is more of a framework since the term "brain" looks like it's a moving target

In another comment I referred to these systems as like comparing hydraulic pumps to human biceps, cars to horses, etc.

We can use the same units of measure, give them the same tasks, but saying they're the same thing only works in the world of poetry

Re: Modern language models refute Chomsky’s approach to language

#48

Post by MAXPOOL »

mcguire wrote: post #6

While I think some of the points in the article are interesting, the usual evidence for the Chomskian approach is the relative lack of input data for learning language by children in the wild. How much input data is used to train modern language models?

I agree. Question has not been settled.

20 year old human has

* heard ~220 million words, talked 50 million words.

* read ~10 million words.

* experienced 420 million seconds of wakeful interaction with the environment (can be used to estimate the limit to conscious decisions, or number of distinct 'epochs' we experience)

From a machine learning perspective human life is surprisingly small set of inputs and actions, just a blip of existence.

Re: Modern language models refute Chomsky’s approach to language

#49

Post by squeaky-clean »

meepmorp wrote: post #39
enord wrote: post #22

Earlier quoted context omitted.

We did, at least once.

No, humans learn their languages through observation and generation - watching older people speak, and then imitating it. They get corrections, too, when they mispronounce or misuse words.

They mean at one point there were no older people to listen to in order to learn language because language did not exist yet. We somehow transitioned from grunts to words.

Re: Modern language models refute Chomsky’s approach to language

#50

Post by jackmott42 »

CydeWeys wrote: post #12
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

Man I remember when people downplayed AlphaGo because it didn't teach itself unsupervised. "Nothing to see here". Only took them a few months to do AlphaZero.
Post reply on HN