Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

211–220 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#211

Post by pfdietz »

FabHK wrote: post #108
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

Chomsky: Birds fly by flapping their wings in a specific way while changing the angle in order to create lift and propulsion. This paper: Planes fly, but don’t flap their wings, ergo Chomsky is wrong.

Chomsky was saying specific things had to be in the brain because it was impossible to do things otherwise. LLMs shoot this argument down even if they aren't how the brain does it.

Re: Modern language models refute Chomsky’s approach to language

#212

Post by mcguire »

8note wrote: post #200
MAXPOOL wrote: post #198

Earlier quoted context omitted.

> some evolved language structures in the brain. That's Chomsky's argument. A small set of constraints for organizing language.

This is testable, right? Put a kid from one language tradition in a spot with a different language tradition, and they won't be able to learn it. Eg. kids with native mandarin speaking parents adopted to native Indo-European parents fail at learning English, and will be better at learning Mandarin than their peers with Indo-European heritage.

That doesn't work. Children will learn whatever language they are exposed to. That is why Chomsky and others talk in terms of a universal grammar.

Re: Modern language models refute Chomsky’s approach to language

#213

Post by meepmorp »

enord wrote: post #201
meepmorp wrote: post #179

Earlier quoted context omitted.

I have no idea what you're talking about and it doesn't really matter to me anyway. Enjoy your day.

Haha I love how you stay so polite while being a pompous a#$. A real double bind! I would apologise for being obtuse but I must say the Gloves Are Off. Good day to you sir!

ok thanks

Re: Modern language models refute Chomsky’s approach to language

#214

Post by somewhereoutth »

refulgentis wrote: post #89
somewhereoutth wrote: post #74

Earlier quoted context omitted.

Exactly. A very uncomfortable truth for those heavily invested (time/money/credence) in this latest AI wave.

It’s odd to see people doomwaving two general reasoning engines. It’s especially hard to parse a dark sweeping condemnation based on…people are investing in it? It doesn’t have the right to assign names to things? Idk what the argument is. My most charitable interpretation is “it cant reason abour anything unless we already said it” which is obviously false.

The heavy investment is what makes this truth uncomfortable - it does not make this truth true (or false).

The point is not so much that we already said it, more that the patterns it encodes and surfaces when prompted are patterns in the written corpus, not of the underlying reality (which it has never experienced). Much like a list of all the addresses in the US (or wherever) will tell you very little about the actual geography of the place.

Re: Modern language models refute Chomsky’s approach to language

#215

Post by dllthomas »

EGreg wrote: post #170
dllthomas wrote: post #125

Earlier quoted context omitted.

It's been a while since I've engaged with the ideas, but it's my understanding that Chomsky was discussing the ability for children to learn the language of the society they're raised in. LLMs are not cheating at that task in the manner you describe. They do seem to need a significantly larger corpus, though, so it's not clear that it actually refutes Chomsky.

And let’s not forget that they are trained on a giga cluster of machines! I am not sure how brute forcing a chess using Monte Carlo Tree Search, or solving Checkers via exhaustive search, would refute a theory about how people with efficient, low-power-consumption brains that grow organically, are able to master Chess.

Eh, it's certainly true that we're throwing tremendously more hardware and power at the problem with an LLM than with a toddler; that's only relevant to whether LLMs refute Chomsky to whatever degree his argument relied on hardware or power consumption (explicitly or implicitly) and my impression is that it didn't.

Re: Modern language models refute Chomsky’s approach to language

#216

Post by bbor »

williamcotton wrote: post #205
bbor wrote: post #135

Earlier quoted context omitted.

…he has? Isn’t his modern term “minimalism”, where he tries to simplify things as much as possible? Regardless, continuing to study the field in no way implies that he’s backed down or meaningfully evolved his basic theories of Universal Grammars. He’s very much still confident in them. Re: predictive processing, in what way does that relate to language…? Even if you apply it to language in a way not mentioned in the…

Re: building a bird… I didn’t come up with the analogy and if we all want to stop talking about building birds I would be the first to agree. But I can’t help by think about all of the whacky ideas like antigravity vital forces that biologists contrived to explain how birds could fly and that it took Bernoulli and the rigorous study of those principles that led to the airfoil… which is how birds actually soar through…

You got me, I believe in the most cited living academic’s theories on cognition to stick it to Elon Musk. So glad you used your geeky principles of discovery to divine that one…

Re: Modern language models refute Chomsky’s approach to language

#217

Post by FabHK »

pfdietz wrote: post #211
FabHK wrote: post #108

Earlier quoted context omitted.

Chomsky: Birds fly by flapping their wings in a specific way while changing the angle in order to create lift and propulsion. This paper: Planes fly, but don’t flap their wings, ergo Chomsky is wrong.

Chomsky was saying specific things had to be in the brain because it was impossible to do things otherwise. LLMs shoot this argument down even if they aren't how the brain does it.

That’s a valid argument that you expressed in two short sentences, yet the paper fails to make in the abstract or first 12 pages.

Re: Modern language models refute Chomsky’s approach to language

#218

Post by EGreg »

dllthomas wrote: post #215
EGreg wrote: post #170

Earlier quoted context omitted.

And let’s not forget that they are trained on a giga cluster of machines! I am not sure how brute forcing a chess using Monte Carlo Tree Search, or solving Checkers via exhaustive search, would refute a theory about how people with efficient, low-power-consumption brains that grow organically, are able to master Chess.

Eh, it's certainly true that we're throwing tremendously more hardware and power at the problem with an LLM than with a toddler; that's only relevant to whether LLMs refute Chomsky to whatever degree his argument relied on hardware or power consumption (explicitly or implicitly) and my impression is that it didn't.

Wait so then what is his argument? Because you can always postulate that a large enough computer can simulate every human and therefore can learn stuff too — thus you don’t need a human to learn language, nyeh!

Obviously, all that stuff ChatGPT says about feelings and emotions came from humans writing it!

Re: Modern language models refute Chomsky’s approach to language

#219

Post by pepa65 »

famouswaffles wrote: post #168
version_five wrote: post #142

Earlier quoted context omitted.

A kind of corollary that I'm sure others have thought of: if llms are so smart and human thought is nothing more than a big language model, why can't they (llms) make up their own training data. Any discussion about how they are "thinking" the way we do is BS, I don't know how so many people who know better have been conned.

Haha what ? They can make training data just fine. https://arxiv.org/abs/2305.07759

Only after they have been fed human language, which was the point.

Re: Modern language models refute Chomsky’s approach to language

#220

Post by pepa65 »

raverbashing wrote: post #84
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

> Also, the LLMs are cheating! They learned from us. Ding ding ding, right answer! Now I leave you with a thought: given that the AI is learning from us, is it possible to have an AI that's much more intelligent than their "source of truth"?

Well, that is answered by AlphaZero (but not for LLM machines).
Post reply on HN