Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

181–190 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#181
post #153
post #9

Earlier quoted context omitted.

Do children really lack input data though? Human sensory input is quite a lot of data. Our languages may have common structure because that structure reflects causality and physics encountered by directly sampling the real world.

Your hypothesis is plausible, but less likely than Chomsky’s IMO. One basic version of the typical chomskian response to this point is: many animals have the same input data, how come were the only ones who have evolved any capacity for language at all? The very best animals at language are apes, and we have to successfully teach one concepts that humans learn while still wearing diapers

Why can't a small neural network do what LLMs do? Same reason animals can't learn human language. They don't have the capacity.

LLMs started getting interesting "all the sudden" when they hit a certain scale, just like biological brains.

Re: Modern language models refute Chomsky’s approach to language

#182
post #144

Earlier quoted context omitted.

They[1] do learn some of it, but a fraction so small it doesn't change the argument. [1]: at least the mamal pets, not goldfishes

That’s a hot take - they learn the meaning of some sounds, but I’d say that’s on a whole different order than human language. Many, many animals communicate concepts using sounds

You shouldn't dismiss the abilities of our fellow mammals too quickly. Of course they are very far from equal to us, but there are several species of mammals that have showed non-trivial use of language, it's not too much of a stretch to imagine that all mammals have at least very basic language building blocks in their brain, even if vestigial.

Re: Modern language models refute Chomsky’s approach to language

#184
post #129

Earlier quoted context omitted.

And how exactly is the structure of the world transformed into a structure of language? Chomsky would say through a universal grammar, which is a framework through which you can set up ways to encode the structure in the world as words.

This implies that the combinatorial space of possible grammars is large. If it isn't, then the structure of the universe would imply a narrow set of possible workable grammars. The combinatorial space of languages is obviously infinite, but is this so for grammars? If not then you would expect many languages to share the same grammars. Seems analogous to the same question about mathematics. Are there many different p…

[deleted]

Re: Modern language models refute Chomsky’s approach to language

#185
post #135

Earlier quoted context omitted.

Chomsky has been adding parameters to his theory to handle exceptions in a way that mimics the endless series of conditional statements appended to knowledge systems of yore. In neuroscience, predictive processing has gained immense favor and can explain language in ways that have nothing to do with innate grammar. https://en.wikipedia.org/wiki/Predictive_coding Exactly how well did "building a bird" work for buildin…

…he has? Isn’t his modern term “minimalism”, where he tries to simplify things as much as possible? Regardless, continuing to study the field in no way implies that he’s backed down or meaningfully evolved his basic theories of Universal Grammars. He’s very much still confident in them. Re: predictive processing, in what way does that relate to language…? Even if you apply it to language in a way not mentioned in the…

Gina Kuperberg at Harvard has done extensive work on how predictive coding operates in language processing. Her lab has explored how people generate predictions during language comprehension and how this is reflected in neural activity.

https://projects.iq.harvard.edu/kuperberglab/people/gina-r-k...

Here's some relevant research:

https://projects.iq.harvard.edu/kuperberglab/publications/dy...

I'll point out that neuroscientists have yet to find the "generative grammar" part of the brain but have seen evidence of a very large network of neurons...

Re: Modern language models refute Chomsky’s approach to language

#186
post #180

Earlier quoted context omitted.

The brain is obviously pre-disposed to language. But evolution fueled learning is by no means from "thin air".

That’s a red herring. Ex post facto the activity of language use as a construction must supersede some enaction by capable actors . This is a complete tautology in terms, and any dispute must therefore be over the ontology in which it is expressed. To elaborate how something came to be is not required to express that something is .

What makes capable actors capable in the first place is not derived from thin air. There's a reason a capable actor came to be where there was none generations prior. That reason did not come about without reaction to some outside vector.

Evolution optimized for language learning abilities before the full capabilities came. Evolution is not thin air.

Re: Modern language models refute Chomsky’s approach to language

#187
post #164

Why is everything these days revolve around ChatGPT(etc). You don't need LLMs to refute Chomsky language models. Modern linguistics pretty much rejected [1] his theories on the basis of evidence. [1] https://www.scientificamerican.com/article/evidence-rebuts-c...

Thanks for posting, finally some support for his supposed debunking! Interesting reading for sure. That work fails to support Chomsky’s assertions. The research suggests a radically different view, in which learning of a child’s first language does not rely on an innate grammar module. Instead the new research shows that young children use various types of thinking that may not be specific to language at all—such as…

I'm reminded of a 400 leve linguistics class I took in undergrad. We had just read Chomsky's Remarks on Nominalization, and one of my classmates remarked, "I don't think this Chomsky guy understands X-Bar theory". The joke being that Chomsky was the major developer of X-Bar theory. We were just reading an early work of his.

This also reminds me of evolution. Some people looked at discoveries in epigenetics and declared that it disproved Darwinian evolution in favor of Lamarckian evolution.

Sure, Darwin's theory of natural selection combined with random variation at the point of reproduction does not explain 100% of evolution, but it is still covers most of it.

Re: Modern language models refute Chomsky’s approach to language

#188
Savage, but maybe fair. If one of Chomsky's underlying claims is indeed that language requires innate hard parsing rules and can't just be derived from probabilistic sampling of a bunch of data - that seems completely dead in the water.

It is entirely likely that the way we operate is probability-first, only deriving rules loosely after taking in lots of experiential data to speed up and simplify that initial fake-it-til-you-make-it understanding. The fact LLMs can get the quality we see using just this approach is a strong indicator that this method of understanding may be a fundamental approach of biological systems too.

(and if you're arguing this is unfair because humans created the language that's being used for probabilistic training - well, look at image models trained on photographs instead and tell me those aren't an example of extreme quality derived purely from mass-inferenced data. Rules-based architectures don't necessarily need apply.)

But honestly, this seems like a silly claim to begin with if it really was claimed. We have formal language theory complexity classes of probabilistic algorithms for a reason - they work! It shouldn't be surprising that the model can stretch down to the fundamentals too. Far fewer programmers (and linguists) were raised to think with these models than deterministic rules-based ones, but the field has been progressing alongside for decades, and now they get to play with powerful LLMs that take probabilistic inferencing to the extreme and will likely prove it works (very elegantly) for everything. This shouldn't be surprising in retrospect.

Chomsky may very well be right that there always exists some fundamental elegant formula underlying any phenomenon (or at least any language). But it's undeniable at this point that simplistic statistical approaches can be applied at scale to those phenomenon and derive highly-useful general models, which also will very likely converge upon the elegant formulas he envisioned. The two are intrinsically linked, neither inseparable.

Re: Modern language models refute Chomsky’s approach to language

#189
post #159

Earlier quoted context omitted.

Of course it is; it’s just a comment on a social media forum. There’s just as little science language motivates me to work. Most of the language society relies on is hallucinations; fiat currency, nation states, constructs like “Senate” and Congress, corporatism, brands, copy-paste of historical terminology, not evidence they’re immutable features of reality. What we recite has nothing to do with what we are. I find…

Are you a large language model?

[dead]

Re: Modern language models refute Chomsky’s approach to language

#190

Earlier quoted context omitted.

I mean have you seen the whole "encode a poem in made up emoji" followed by "decode the poem from emoji" stuff? I think it's not unreasonable to think with the right prompts LLMs could do this.

The LLMs that are doing this are trained on trillions of examples of human language. This is not remotely a counterexample. Now if an LLM can invent a new full language from scratch, without any training data of existing languages (like how AlphaZero learned to play Go), then that will be impressive, and a difference in kind.

"In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened."
Post reply on HN