Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

121–130 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#121

Post by refulgentis »

root_axis wrote: post #109
refulgentis wrote: post #89

Earlier quoted context omitted.

It’s odd to see people doomwaving two general reasoning engines. It’s especially hard to parse a dark sweeping condemnation based on…people are investing in it? It doesn’t have the right to assign names to things? Idk what the argument is. My most charitable interpretation is “it cant reason abour anything unless we already said it” which is obviously false.

> one of which is an average 14 year old, the other an honors student college freshman The point is that they're not those things. Yes, language models can produce solutions to language tests that a 14 year old could also produce solutions for, but a calculator can do the same thing in the dimension of math - that doesn't make a calculator a 14 year old.

Yes, the AI isn’t literally a 14 year old, and we should do an anthromorphization acknowledgement. Thank you for pointing it out, it can waste a lot of time when you get sloppy with language in AI discussions.

I removed the reference, in retrospect, it’s unnecessary. No need to indicate the strong performance, we’re all aware.

Re: Modern language models refute Chomsky’s approach to language

#122

Post by azakai »

PeterCorless wrote: post #117

The LLMs don't understand anything. At all. They have rules they respond to. But that doesn't mean they actually "get" a single word of what they're blathering. A computer has no sense of "intent" or "meaning" to what they have indexed and scanned. There is no intent or meaning to what they spit out in generated text. We're back in the Chinese room. https://en.wikipedia.org/wiki/Chinese_room

The Chinese Room Argument is considered by many to be incorrect on those matters. But it has led to decades of interesting philosophical debate, which is worth reading to see the various perspectives. This is an excellent summary:

https://plato.stanford.edu/entries/chinese-room/

Re: Modern language models refute Chomsky’s approach to language

#123

Post by meepmorp »

enord wrote: post #53
meepmorp wrote: post #39

Earlier quoted context omitted.

No, humans learn their languages through observation and generation - watching older people speak, and then imitating it. They get corrections, too, when they mispronounce or misuse words.

I didn’t say everyone learns it from scratch at least one, I said we did.

That's not learning, though, is it? Inventing, maybe, or developing or evolving.

Re: Modern language models refute Chomsky’s approach to language

#124

Post by GuB-42 »

neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

We could use programming languages as a counterpoint.

LLMs can code in the same way they can use natural languages. But we know that programming languages have structure, we made them that way, from scratch, using Chomsky's theory no less.

Saying that because LLMs can learn programming languages using a different approach and therefore disprove the very theory they are built on is absurd.

Anyways, the paper is long and full of references, I didn't analyse it, does it include looks inside the model? For example, for LLMs to write code correctly, the structure of programming languages must be encoded somewhere in the weights of the model. A way to more convincingly disprove Chomsky's ideas would be to find which part of the network encodes structure in programming languages, and show that there is nothing similar for natural languages.

Re: Modern language models refute Chomsky’s approach to language

#125

Post by dllthomas »

CydeWeys wrote: post #12
neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

It's been a while since I've engaged with the ideas, but it's my understanding that Chomsky was discussing the ability for children to learn the language of the society they're raised in. LLMs are not cheating at that task in the manner you describe.

They do seem to need a significantly larger corpus, though, so it's not clear that it actually refutes Chomsky.

Re: Modern language models refute Chomsky’s approach to language

#126

Post by QuadmasterXLII »

famouswaffles wrote: post #18
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Individual humans aren’t, but we’re talking about the emergent properties of a swarm of humans. 6 gigabytes of dna describes an entity that, if copied 60 times and dropped on an island, will produce a viable language in inly a few decades. We haven’t found a NN architecture with this property for language, but we have for , eg, chess, starcraft, and go

Re: Modern language models refute Chomsky’s approach to language

#127

Post by anthonybsd »

Why is everything these days revolve around ChatGPT(etc). You don't need LLMs to refute Chomsky language models. Modern linguistics pretty much rejected [1] his theories on the basis of evidence.

[1] https://www.scientificamerican.com/article/evidence-rebuts-c...

Re: Modern language models refute Chomsky’s approach to language

#128

Post by pas »

taeric wrote: post #46
zvmaz wrote: post #29

From the conclusion: > First, the fact that language models can be trained on large amounts of text data and can generate human-like language without any explicit instruction on gram- mar or syntax suggests that language may not be as biologically determined as Chomsky has claimed. Instead, it suggests that language may be learned and developed through exposure to language and interactions with others. I'm not a ling…

It contradicts the idea that you have to teach language using syntax and grammar. Which... I confess I thought was already not believed? We certainly aren't teaching kids in the home how to decline and conjugate words. Are we? (Similarly, languages that have gender are typically just picked up by usage, not necessarily ingrained by reasoning. Which leads to the obvious bad results when people think that there was sol…

yep. Chomsky used the term "universal grammar", but it's much more likely a universal abstract semantics++ thing (coupled with sound processing stuff, plus repetition, plus a bunch of other stuff in the brain that helps learning in general).

how much is innate, what exactly does that mean, all good questions. and of course raw intelligence (pattern matching, strategizing, learning, adaptiveness, modeling, ability to form a sort of consistent and goal-orientedly useful predictive model of the world based on inputs, and goal-oriented control of behavior based on these aforementioned models) by definition can learn language.

and of course it's a strange question of are LLMs intelligent in this sense despite lacking goals?

Re: Modern language models refute Chomsky’s approach to language

#129

Post by api »

sudosysgen wrote: post #16
api wrote: post #9

Earlier quoted context omitted.

Do children really lack input data though? Human sensory input is quite a lot of data. Our languages may have common structure because that structure reflects causality and physics encountered by directly sampling the real world.

And how exactly is the structure of the world transformed into a structure of language? Chomsky would say through a universal grammar, which is a framework through which you can set up ways to encode the structure in the world as words.

This implies that the combinatorial space of possible grammars is large. If it isn't, then the structure of the universe would imply a narrow set of possible workable grammars.

The combinatorial space of languages is obviously infinite, but is this so for grammars? If not then you would expect many languages to share the same grammars.

Seems analogous to the same question about mathematics. Are there many different possible arithmetics? No. There are infinitely many ways to express arithmetic symbolically but there is only one arithmetic. 2 + 2 never equals 5.

Re: Modern language models refute Chomsky’s approach to language

#130

Post by croes »

Joeri wrote: post #94
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

I just asked an LLM to create a language and provide a demonstration and this is what it said. Call it a stochastic parrot if you want, but I’m pretty sure a linguist can prompt it to properly invent a language. Sure, I can invent a new language for you! Let's call it "Vorin" for the purposes of this demonstration. Vorin is a tonal language with a complex system of noun classes and a relatively simple verb conjugatio…

This sounds like a backwards explanation to explain why the same words have different meanings.

Maybe the LLM should write the tones too, so it sounds like a child inventing a language and when you point out its logical inconsistencies it invents new rules to fit.

Post reply on HN