Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

191–200 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#191

Post by shaunxcode »

janalsncm wrote: post #40
shaunxcode wrote: post #20

Earlier quoted context omitted.

Thank you! It is like arguing that a human engaging in the creation of a landscape portrait using the traditional method of oil painting has been "refuted" by a computer generating vector graphics from statistical descriptions of the same scene. Both yield art but they are clearly different in interesting ways. Neither refutes or outmodes the other. Or maybe I'm wrong and actually trees have refuted mushrooms!

Since large fungi came first, maybe trees have refuted the mushrooms? https://en.m.wikipedia.org/wiki/Prototaxites

That is my point though: the notion of one plant refuting another is absurd if they both still thrive. Likewise the underpinnings of how LLMs work does not refute the conceptual framework of universal grammar. Especially as refutation means to disprove (not improve, increment, evolve or augment). One more analogy: jiu jitsu does not refute aikido even if it is better in an MMA context. When reaching for refutation prefer synthesis.

Re: Modern language models refute Chomsky’s approach to language

#192

Post by williamcotton »

gizmo686 wrote: post #141
williamcotton wrote: post #111

Earlier quoted context omitted.

Chomsky has been adding parameters to his theory to handle exceptions in a way that mimics the endless series of conditional statements appended to knowledge systems of yore. In neuroscience, predictive processing has gained immense favor and can explain language in ways that have nothing to do with innate grammar. https://en.wikipedia.org/wiki/Predictive_coding Exactly how well did "building a bird" work for buildin…

Chomsky is not trying to build a flying machine. He is trying to study birds. Biologists don't look at a 737 and say "that's obviously the way flight works, I wonder where the engine on that seagull is".

The only reason I was talking about "building a bird" in the first place is because that is what the person I was responding to was talking about. At this point I don't even understand what this analogy is supposed to be. Is the bird supposed to be "language"? Or "humans"? Or "human language"? What's the point of this study, just to chit-chat about things? What proof do we have the Chomsky's theories of language are true? Neuroscience has actual research that backs the claims of predictive coding and language acquisition and comprehension! Some names in case you're curious: Gina Kuperberg, Ellen Lau, Florian Jaeger and Roger Levy, just to name a few of the more influential researchers.

Re: Modern language models refute Chomsky’s approach to language

#193

Post by throwawaymaths »

mempko wrote: post #80
williamcotton wrote: post #69

Earlier quoted context omitted.

What about the fact that trying to use Chomsky's approach to universal grammar does not result in working language models yet a statistical approach does? The Norvig-Chomsky debate is kind of old at this point: https://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fig...

Because Chomsky is trying to build a bird, and Norvig is trying to build an airplane. It's much easier to build an airplane to fly than a bird. Chomsky is trying to explain how humans create language. LLM are creating language, but not the way humans do. Nothing about this paper refutes Chomsky's claims.

> It's much easier to build an airplane to fly than a bird.

Not really. You could probably put together a flapping bird toy (can buy these mass manufactured too) in about a couple of months of trial and error. Not quite as sophisticated as feathers but the principles are the same. You probably couldn't build an airplane.

Re: Modern language models refute Chomsky’s approach to language

#194

Post by williamcotton »

MrBuddyCasino wrote: post #15
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

Wordcels think LLMs imitate the human brain, when a shape rotator knows they really just imitate human language.

Wouldn’t the wordcels be the Chomsky generative grammar supporters and the shape rotators the neuroscientists who support a statistical approach?

Re: Modern language models refute Chomsky’s approach to language

#195

Post by williamcotton »

azakai wrote: post #118
zvmaz wrote: post #100

Earlier quoted context omitted.

Also, dogs and cats, and even our close relatives, primates, don't develop a capacity to language.

Yes, and that's one of my favorite Chomsky points. I don't remember the exact phrasing, but something like: "Language is innate in humans because in every household practically all children learn it while none of the pets do."

Well how large are the neural networks in both humans and cats? And haven’t we seen empirical evidence that some astonishing emergent behavior is exhibited by large artificial neural networks?

Re: Modern language models refute Chomsky’s approach to language

#196

Post by AbrahamParangi »

username332211 wrote: post #102
AbrahamParangi wrote: post #38

Earlier quoted context omitted.

In order to believe this, you'd need to be able to imagine a specific test of something that an LLM could not do under any circumstances. Previously, that test could have been something like "compose a novel sonnet on a topic". Today, it is much less clear that such a test (that won't be rapidly beaten) even exists.

You could use a Markov chain to generate poetry with rhyme and meter[1]. Granted, it wouldn't be a very good one, but that just makes an LLM a refinement to older probabilistic methods. As for something LLMs are unlikely to do under any circumstances, there's already a fairly obvious example. They can't keep a secret, hence prompt injections. [1] https://us.pycon.org/2020/schedule/presentation/112/

Do you really believe that an LLM that can keep a secret cannot be made? I suspect that we could do this trivially and the "LLMs can't keep a secret" is a specific product of finetuning for helpfulness.

Re: Modern language models refute Chomsky’s approach to language

#197

Post by 8note »

mjburgess wrote: post #172
dTal wrote: post #113

Earlier quoted context omitted.

There's nothing qualitatively less "in the world" about a language model than a human. Yes, a human has more senses, and is doubtless exposed to huge categories of training data that a language model doesn't have access to - but it's false to draw a sharp dichotomy between knowing what an iPhone looks like, and knowing how people talk about iPhones. Consider two people - one, a Papau New Guinea tribesperson from a pr…

LLMs don't have any senses, not merely fewer. LLMs don't have any concepts, not merely named ones. A concept is a sensory-motor technique abstracted into a pattern of thought developed by an animal, in a spatio-temporal environment, for a purpose. LLMs are just literally an ensemble of statistical distributions over text symbols. In generating text, they're just sampling from a compressed bank of all text ever digiti…

LLMs have two senses, time and text

Re: Modern language models refute Chomsky’s approach to language

#198

Post by MAXPOOL »

IshKebab wrote: post #139
MAXPOOL wrote: post #48

Earlier quoted context omitted.

I agree. Question has not been settled. 20 year old human has * heard ~220 million words, talked 50 million words. * read ~10 million words. * experienced 420 million seconds of wakeful interaction with the environment (can be used to estimate the limit to conscious decisions, or number of distinct 'epochs' we experience) From a machine learning perspective human life is surprisingly small set of inputs and actions,…

LLMs learn from unlabeled data. Children definitely do not. There's a huge difference. I would not be surprised if LLMs could learn a lot more efficiently if they had carefully constructed training data, with video and sound. But also humans have been speaking for so long it's silly to imagine we don't have some evolved language structures in the brain. I don't know why anyone would single that out for skepticism whi…

> some evolved language structures in the brain.

That's Chomsky's argument. A small set of constraints for organizing language.

Re: Modern language models refute Chomsky’s approach to language

#199

Post by 8note »

CydeWeys wrote: post #106
spullara wrote: post #19

Earlier quoted context omitted.

Every human alive is cheating, they learned from us!

Humanity invented language though. LLMs did not; they're merely parroting our language back to us.

Wed punish them in training if they came up with their own language

Re: Modern language models refute Chomsky’s approach to language

#200

Post by 8note »

MAXPOOL wrote: post #198
IshKebab wrote: post #139

Earlier quoted context omitted.

LLMs learn from unlabeled data. Children definitely do not. There's a huge difference. I would not be surprised if LLMs could learn a lot more efficiently if they had carefully constructed training data, with video and sound. But also humans have been speaking for so long it's silly to imagine we don't have some evolved language structures in the brain. I don't know why anyone would single that out for skepticism whi…

> some evolved language structures in the brain. That's Chomsky's argument. A small set of constraints for organizing language.

This is testable, right?

Put a kid from one language tradition in a spot with a different language tradition, and they won't be able to learn it.

Eg. kids with native mandarin speaking parents adopted to native Indo-European parents fail at learning English, and will be better at learning Mandarin than their peers with Indo-European heritage.

Post reply on HN