Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

Post reply on HN
Print view View original

111–120 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#111

Post by williamcotton »

mempko wrote: post #80
williamcotton wrote: post #69

Earlier quoted context omitted.

What about the fact that trying to use Chomsky's approach to universal grammar does not result in working language models yet a statistical approach does? The Norvig-Chomsky debate is kind of old at this point: https://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fig...

Because Chomsky is trying to build a bird, and Norvig is trying to build an airplane. It's much easier to build an airplane to fly than a bird. Chomsky is trying to explain how humans create language. LLM are creating language, but not the way humans do. Nothing about this paper refutes Chomsky's claims.

Chomsky has been adding parameters to his theory to handle exceptions in a way that mimics the endless series of conditional statements appended to knowledge systems of yore.

In neuroscience, predictive processing has gained immense favor and can explain language in ways that have nothing to do with innate grammar.

https://en.wikipedia.org/wiki/Predictive_coding

Exactly how well did "building a bird" work for building flying machines? Birds use the same principle as a fixed wing when it comes to soaring flight. "Building a bird" without the principles of an airfoil and just mimicking the flapping wings does not result in flight.

Re: Modern language models refute Chomsky’s approach to language

#113

Post by dTal »

mjburgess wrote: post #41
famouswaffles wrote: post #18

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans learn from the structure of the world -- not the structure of language. LLMs cheat at generating text because they do so via a model of the statistical structure of text. We're in the world , it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily . What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all lik…

There's nothing qualitatively less "in the world" about a language model than a human. Yes, a human has more senses, and is doubtless exposed to huge categories of training data that a language model doesn't have access to - but it's false to draw a sharp dichotomy between knowing what an iPhone looks like, and knowing how people talk about iPhones.

Consider two people - one, a Papau New Guinea tribesperson from a previously uncontacted tribe who is allowed to handle a powered-down iPhone, and told it is an "iPhone", but is otherwise ignorant of its behavior - the other, a cross-platform mobile software developer who has never actually held a physical iPhone, but is intimately familiar with its build systems, API, cultural context etc. Between the two of them, who better understands what an iPhone "is"?

You make a good point about inventing words to refer to new concepts. There's nothing theoretically stopping a language model from identifying some concept in its training data that we don't have a word for, inventing a word for it, and using it to give us a perspective we hadn't considered. It would be very useful if it did that! I suspect we don't tend to see that simply because it's a very rare occurrence in the text it was trained on.

Re: Modern language models refute Chomsky’s approach to language

#114

Post by npinsker »

Joeri wrote: post #94
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

I just asked an LLM to create a language and provide a demonstration and this is what it said. Call it a stochastic parrot if you want, but I’m pretty sure a linguist can prompt it to properly invent a language. Sure, I can invent a new language for you! Let's call it "Vorin" for the purposes of this demonstration. Vorin is a tonal language with a complex system of noun classes and a relatively simple verb conjugatio…

I’m pretty sure of the opposite. Even for your minimal example, the translations seem very suspect (what’s the word for “working” in Vorin?).

Re: Modern language models refute Chomsky’s approach to language

#115

Post by OskarS »

gizmo686 wrote: post #91
chubot wrote: post #52

Earlier quoted context omitted.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

For obvious ethical reasons, we have never perfectly conducted this experiment, but Nicaraguan Sign Language comes close. The government created a school for deaf children who did not have a common language. Many of the children did not know any language prior to attending. Although, they did have "home sign" systems, which are basic signing systems spontaneously developed between deaf children and their speaking car…

The common thread in both cases is (as you say) children: in both creoles and NSL, the early versions of the language are somewhat primitive (being developed by older children and adults), but when children are exposed to it from an early age, the languages acquire much more complexity and sophistication, becoming every bit as rich as "regular" languages. It fits well with the hypothesis that the early years of life are crucial for language development, and that there is indeed something innate in most humans for deep structural understanding of language. It's not just statistical pattern recognition.

Re: Modern language models refute Chomsky’s approach to language

#116

Post by aezart »

jackmott42 wrote: post #51
asdff wrote: post #33

Earlier quoted context omitted.

Imagine being told all you need to do to learn Spanish, is to read a 300,000 word Spanish dictionary end to end so that you can probalistically come up with 1000 conversational phrases. Anyone who has learned a language can tell you it just doesn't work like that. You don't work by accumulating a massive dataset and training on it. No one can hold such a massive dataset of anything in their head at once.

The LLM doesn't hold the whole dataset in its head either.

Not for a lack of trying.

If you give it a small enough training set or a big enough neural network, it will directly memorize the whole thing. You have to intentionally make its brain too small to do that in order to force it to find patterns in the data instead.

Re: Modern language models refute Chomsky’s approach to language

#117

Post by PeterCorless »

The LLMs don't understand anything. At all. They have rules they respond to. But that doesn't mean they actually "get" a single word of what they're blathering.

A computer has no sense of "intent" or "meaning" to what they have indexed and scanned. There is no intent or meaning to what they spit out in generated text.

We're back in the Chinese room. https://en.wikipedia.org/wiki/Chinese_room

Re: Modern language models refute Chomsky’s approach to language

#118

Post by azakai »

zvmaz wrote: post #100
gizmo686 wrote: post #59

This paper completly misses the point of linguistics as a discipline that generative grammar operates in. We have known that it is possible to understand language without innateness. That is what linguists do. If you look at how linguists know about innate features, the answer is almost always by first discovering them explicitly while analysing language data; not by opening a brain to see what is innately inside. [0…

Also, dogs and cats, and even our close relatives, primates, don't develop a capacity to language.

Yes, and that's one of my favorite Chomsky points. I don't remember the exact phrasing, but something like:

"Language is innate in humans because in every household practically all children learn it while none of the pets do."

Re: Modern language models refute Chomsky’s approach to language

#119

Post by riku_iki »

neosat wrote: post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way.

we actually don't know what is inside LM too, so it is possible LM statistically learns syntax and semantics, and it is major part of output quality.

Re: Modern language models refute Chomsky’s approach to language

#120

Post by lsy »

Joeri wrote: post #94
CydeWeys wrote: post #12

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

I just asked an LLM to create a language and provide a demonstration and this is what it said. Call it a stochastic parrot if you want, but I’m pretty sure a linguist can prompt it to properly invent a language. Sure, I can invent a new language for you! Let's call it "Vorin" for the purposes of this demonstration. Vorin is a tonal language with a complex system of noun classes and a relatively simple verb conjugatio…

None of this makes sense.

1."lè" with a low tone means fine, but "nì síkòní" means "I'm fine", while "là síkòní" means "he/she/it is fine"?

2. The example of "noun prefixes" doesn't include different noun prefixes?

3. The example of "verb ending changes" doesn't include different verb endings?

4. Since the second clause of the second phrase apparently means "I'm fine", the first word must mean "you", but then the second word is repeated in the next sentence which has nothing to do with thanking someone? And "síkòní" gets sprinkled into three sentences with no common concepts?

This just looks, predictably, like gibberish spit out by an LLM.

Post reply on HN