Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

141–150 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#141

Post by gizmo686 »

williamcotton wrote: post #111
mempko wrote: post #80

Earlier quoted context omitted.

Because Chomsky is trying to build a bird, and Norvig is trying to build an airplane. It's much easier to build an airplane to fly than a bird. Chomsky is trying to explain how humans create language. LLM are creating language, but not the way humans do. Nothing about this paper refutes Chomsky's claims.

Chomsky has been adding parameters to his theory to handle exceptions in a way that mimics the endless series of conditional statements appended to knowledge systems of yore. In neuroscience, predictive processing has gained immense favor and can explain language in ways that have nothing to do with innate grammar. https://en.wikipedia.org/wiki/Predictive_coding Exactly how well did "building a bird" work for buildin…

Chomsky is not trying to build a flying machine. He is trying to study birds.

Biologists don't look at a 737 and say "that's obviously the way flight works, I wonder where the engine on that seagull is".

Re: Modern language models refute Chomsky’s approach to language

#142

Post by version_five »

CydeWeys wrote: post #12
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

A kind of corollary that I'm sure others have thought of: if llms are so smart and human thought is nothing more than a big language model, why can't they (llms) make up their own training data. Any discussion about how they are "thinking" the way we do is BS, I don't know how so many people who know better have been conned.

Re: Modern language models refute Chomsky’s approach to language

#143

Post by bbor »

gizmo686 wrote: post #59

This paper completly misses the point of linguistics as a discipline that generative grammar operates in. We have known that it is possible to understand language without innateness. That is what linguists do. If you look at how linguists know about innate features, the answer is almost always by first discovering them explicitly while analysing language data; not by opening a brain to see what is innately inside. [0…

For a moment I was going to waste my afternoon arguing with people desperately predisposed to being the underdog in the fight against the Father of Linguistics, but you’ve said everything I ever could beautifully, so if this doesn’t help nothing will. Especially love the last paragraph, clarified that pattern for me.

On a lighter note, I do expect “Modern Language Models Refute…” to be the new “All you need is…”! It’s just too provocative not to click on

Re: Modern language models refute Chomsky’s approach to language

#144

Post by bbor »

littlestymaar wrote: post #140
azakai wrote: post #118

Earlier quoted context omitted.

Yes, and that's one of my favorite Chomsky points. I don't remember the exact phrasing, but something like: "Language is innate in humans because in every household practically all children learn it while none of the pets do."

They[1] do learn some of it, but a fraction so small it doesn't change the argument. [1]: at least the mamal pets, not goldfishes

That’s a hot take - they learn the meaning of some sounds, but I’d say that’s on a whole different order than human language. Many, many animals communicate concepts using sounds

Re: Modern language models refute Chomsky’s approach to language

#145

Post by sudosysgen »

justrealist wrote: post #107
romwell wrote: post #81

Earlier quoted context omitted.

>There's really no good data to support this claim. Fine. We know that languages did not exist at some point. We made them exist.

Yes, I acknowledge that we have evidence that groups of tribal humans will evolve language constructs over tens of thousands of years. Whether that happens in 1 generation or 10,000 generations is completely unknown.

As others have mentioned in this thread, the Nicaraguan Sign Language shows it can happen in years, not even generations.

Re: Modern language models refute Chomsky’s approach to language

#146

Post by effed3 »

I remember, some time ago, someone (more than one, to be precise) used NN (neuralnets) to resolve some math problems (differential equations, IIRC, i have lost the refs). Suddently some others stated the death of Mathematics, and in general, of Science as we all know, just pour all data in a NN and solution will appear, no more theory, models, hypothesis, experiments,.. needed. Nothing of this happened, but from time to time this -hallucination- return, missing the difference between Science and Technology, between a machine that work (or appear to work) and a model/theory that explain how/why.

Re: Modern language models refute Chomsky’s approach to language

#147

Post by bbor »

taeric wrote: post #46
zvmaz wrote: post #29

From the conclusion: > First, the fact that language models can be trained on large amounts of text data and can generate human-like language without any explicit instruction on gram- mar or syntax suggests that language may not be as biologically determined as Chomsky has claimed. Instead, it suggests that language may be learned and developed through exposure to language and interactions with others. I'm not a ling…

It contradicts the idea that you have to teach language using syntax and grammar. Which... I confess I thought was already not believed? We certainly aren't teaching kids in the home how to decline and conjugate words. Are we? (Similarly, languages that have gender are typically just picked up by usage, not necessarily ingrained by reasoning. Which leads to the obvious bad results when people think that there was sol…

Yeah perhaps that’s causing confusion for science enthusiasts new to this debate - Chomsky is definitely not talking about what a lay person would call “Grammar”. Your point is one of his main (implicit) supports though: on the face of it, it seems absolutely insane that a child could pick up complex linguistic concepts in a few short years, with many orders of magnitude less data than a LLM needs to reach the same capabilities

Re: Modern language models refute Chomsky’s approach to language

#148

Post by michaelhartm »

michaelhartm wrote: post #79

Nobody knows how the LLMs work under the hood. It's just lots of stacked transformers that encode various concepts. Nothing in this book refutes whether Chomsky's concepts are actually being encoded in LLMs or not. For all we know, Chomsky's concept of "binding principles", "binary branching" etc could be represented inside the inner layers of these many billion parameter models. In fact, I'd argue that this is the r…

Btw, semantics and syntax is separated in the LLMs (the author is wrong). The embedding function (matmul) can map syntax and the proximity in the embedding (e.g. cosine similarity) is the semantics (that's attention). So not convinced. Chomsky might be wrong or right, but this author hasn't proven it.

Re: Modern language models refute Chomsky’s approach to language

#149

Post by relativty »

fooker wrote: post #131
relativty wrote: post #101

Earlier quoted context omitted.

Yeah because all the other things are innate; visual/spatial awareness, touch, smell, vocalization… Humanoids went millions of years literally learning to navigate 3D space and sense “enough heat, food, water” etc Nomadic tribes had built shared resource depots millennia before language. I can see the color gradients of the trees and feel muscles relax without words. Human language beyond some utilitarian labels just…

> 90% of human communication is unspoken. This is as scientific as the idea of humans just using 10% of our brains.

Of course it is; it’s just a comment on a social media forum.

There’s just as little science language motivates me to work. Most of the language society relies on is hallucinations; fiat currency, nation states, constructs like “Senate” and Congress, corporatism, brands, copy-paste of historical terminology, not evidence they’re immutable features of reality.

What we recite has nothing to do with what we are. I find the appeals to non-existent political truisms primate gibberish.

It seems perfectly clear to me many facts of society are just memorized and recited prompt hacks. Language is the goto tool for propagandists, to obscure sensory connection to reality.

There is over 100 years of propaganda research available, too much for me to sort through, but scientific measure of such is not new; new to anyone unaware of it but not to humanity.

Re: Modern language models refute Chomsky’s approach to language

#150

Post by sdenton4 »

zvmaz wrote: post #100
gizmo686 wrote: post #59

This paper completly misses the point of linguistics as a discipline that generative grammar operates in. We have known that it is possible to understand language without innateness. That is what linguists do. If you look at how linguists know about innate features, the answer is almost always by first discovering them explicitly while analysing language data; not by opening a brain to see what is innately inside. [0…

Also, dogs and cats, and even our close relatives, primates, don't develop a capacity to language.

That could be more a question of the sensorium than brain structure, though...

And whales might say something similar about us, that humans have a very complex society despite being seemingly unable to learn to speak coherently.

Post reply on HN