Live data from Hacker News

Modern language models refute Chomsky’s approach to language

scholar.google.com

91–100 of 244 posts

Re: Modern language models refute Chomsky’s approach to language

#91
post #52

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Children raised in isolation will spontaneously develop language I don’t have a reference handy now (someone can probably do better) but I believe one way to see this is via the hearing impaired or hearing and sight impaired

For obvious ethical reasons, we have never perfectly conducted this experiment, but Nicaraguan Sign Language comes close. The government created a school for deaf children who did not have a common language. Many of the children did not know any language prior to attending. Although, they did have "home sign" systems, which are basic signing systems spontaneously developed between deaf children and their speaking care takers.

We can also look at creole languages. When distinct linguistic groups come into contact without a common language they will, in some cases, develop "pidgin" languages to communicate. These languages are created by the adults who are already fluent speakers of their native language. What is interesting is what happens with children born into such communities. They grow up hearing the pidgin language, but they do not learn the pidgin. Instead, the language they learn is a creole. It is based on the pidgin, but more complete and consistent with the way human language works in general. In effect, like all children, they are relying on their innate knowledge of language. The pidgin fills in the language specific parameters, but when the pidgin contradicts an aspect of innate language the children (for the most part), just don't aquire that portion of the language.

Re: Modern language models refute Chomsky’s approach to language

#92
post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

Wordcels think LLMs imitate the human brain, when a shape rotator knows they really just imitate human language.

This sentence made me despise my own literacy.

Re: Modern language models refute Chomsky’s approach to language

#93
> First, the fact that language models can be trained on large amounts of text data and can generate human-like language without any explicit instruction on grammar or syntax suggests that language may not be as biologically determined as Chomsky has claimed

"The fact that this advanced drone released in the year 2080 that can drive with the same agility as an eagle proves that the eagle's flying ability is not as biologically determined as some people claim.

In fact, any organism can fly if it sees enough data about flying!"

Re: Modern language models refute Chomsky’s approach to language

#94
post #8

The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

I just asked an LLM to create a language and provide a demonstration and this is what it said. Call it a stochastic parrot if you want, but I’m pretty sure a linguist can prompt it to properly invent a language.

Sure, I can invent a new language for you! Let's call it "Vorin" for the purposes of this demonstration.

Vorin is a tonal language with a complex system of noun classes and a relatively simple verb conjugation system. It is spoken by the Vorin people, who live in a mountainous region of an imaginary world. Here is a sample conversation in Vorin:

Person 1: Ní lé màrì?

Person 2: Lé vàlà, nì síkòní.

Person 1: Vàlà síkòní?

Person 2: Àmà, síkòní là nì tékàlá.

Translation:

Person 1: How are you?

Person 2: I'm fine, thank you.

Person 1: Are you working?

Person 2: Yes, I'm working right now.

In Vorin, the tone of each word is very important, and can change the meaning of the word entirely. For example, "lé" with a high tone means "you", while "lé" with a low tone means "fine". In addition to tone, Vorin has a system of noun classes, which are indicated by prefixes on the nouns themselves. For example, "màrì" means "you" if it is in the second person singular class, but it can also mean "house" if it is in the third person singular class.

Vorin's verb conjugation system is relatively simple, with only a few tenses and moods. Verbs are also marked for subject agreement, which means that the verb ending changes depending on the subject of the sentence. For example, "nì síkòní" means "I'm fine", while "là síkòní" means "he/she/it is fine".

Overall, Vorin is a complex and interesting language with a rich grammatical structure and unique tonal system.

Re: Modern language models refute Chomsky’s approach to language

#95

Earlier quoted context omitted.

>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.

Humans invented English, and every other human language. Not one single human, and not in a single lifetime, but yes, we did it. LLMs are not doing this.

I mean have you seen the whole "encode a poem in made up emoji" followed by "decode the poem from emoji" stuff? I think it's not unreasonable to think with the right prompts LLMs could do this.

Re: Modern language models refute Chomsky’s approach to language

#96
post #30

Earlier quoted context omitted.

1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universa…

> No Language model is yet even close to the scale of the human brain If GPT-4 has 100 trillion parameters, it has as many parameters as the human brain has synapses. Synapses are a lot simpler than parameters; they're digital. A single neuron needs many synapses, all of roughly equal weight, emitting many pulses over a short time in order to convey a single weighted value. On top of that, you may have heard that the…

There's a single medium.com grifter claiming 100 trillion with nothing to back it up. 1 trillion for the unavailable instruct model and considerably less for the chat model is most likely.

Re: Modern language models refute Chomsky’s approach to language

#97
post #94

Earlier quoted context omitted.

> The fact that you can replicate coherent text from probabilistic analysis and modeling of a very large corpus does not mean that humans acquire and generate language the same way. Also, the LLMs are cheating! They learned from us. It's entirely possible that you do need syntax/semantics/sapience to create the original corpus, but not to duplicate it. Let's see an AlphaZero-style version of an LLM, that learns langu…

I just asked an LLM to create a language and provide a demonstration and this is what it said. Call it a stochastic parrot if you want, but I’m pretty sure a linguist can prompt it to properly invent a language. Sure, I can invent a new language for you! Let's call it "Vorin" for the purposes of this demonstration. Vorin is a tonal language with a complex system of noun classes and a relatively simple verb conjugatio…

You're missing the point though, which is that that LLM was already trained on trillions of examples of human languages. Once it already knows many languages, inventing one more isn't hard. (Though I don't think it's actually "invented" this language to any great depth.)

Let's see it invent language from scratch without being presented any examples of languages to use as training data. That's what humanity did. Can LLMs do it? I doubt it, not yet.

Re: Modern language models refute Chomsky’s approach to language

#98
On Chomsky, here is a recording of some audio from Chomsky where he uses a rhetorical argument for why language may be innate to humans rather than (fully) learned. From 1992. Just 5 minutes.

https://youtu.be/CPgDALpS-7k

Let's make sure what the computer scientists understand Chomsky to have stated is actually aligned. Chomsky didn't say the ONLY way to create language is via the brain. His view, instead, is that evolution programmed language development into the brain -- that it is not learned (entirely) by peer osmosis. That the brain has some structure for language, built-in, which is the unlocked in various ways via socialization.

Summary of Chomsky's view, paraphrased:

"It is a strange intuition [that most other people have]. Above the neck, we insist everything [in human development] comes from experience. Below the neck, we're willing to accept the idea that [...] it comes from inside. [...] But: it is hard to look at the Sun setting and say, it's not 'setting', the Earth is actually turning. Similarly, with people, it's hard for us to look at them and not see them as minds inside bodies. This leads us to this false approach: below the neck, we are willing to pursue the sciences, and if that leads us to believe development is internally programmed, we'll accept it. But above the neck, we'll be completely irrational; we're going to insist on beliefs and explanations we'd never normally dream of in other rational areas."

---

This YouTube clip came to mind, but here is a more detailed explainer from Stanford Encyclopedia of Philosophy:

"Clearly, there is something very special about the brains of human beings that enables them to master a natural language — a feat usually more or less completed by age 8 or so. ... This article introduces the idea, most closely associated with the work of the MIT linguist Noam Chomsky, that what is special about human brains is that they contain a specialized ‘language organ,’ an innate mental ‘module’ or ‘faculty,’ that is dedicated to the task of mastering a language.

On Chomsky's view, the language faculty contains innate knowledge of various linguistic rules, constraints and principles; this innate knowledge constitutes the ‘initial state’ of the language faculty. In interaction with one's experiences of language during childhood — that is, with one's exposure to what Chomsky calls the ‘primary linguistic data’ or ‘pld’ — it gives rise to a new body of linguistic knowledge, namely, knowledge of a specific language (like Chinese or English). This ‘attained’ or ‘final’ state of the language faculty constitutes one's ‘linguistic competence’ and includes knowledge of the grammar of one's language. This knowledge, according to Chomsky, is essential to our ability to speak and understand a language (although, of course, it is not sufficient for this ability: much additional knowledge is brought to bear in ‘linguistic performance,’ that is, actual language use)."

source: https://plato.stanford.edu/entries/innateness-language/

Re: Modern language models refute Chomsky’s approach to language

#99
post #9
post #6

While I think some of the points in the article are interesting, the usual evidence for the Chomskian approach is the relative lack of input data for learning language by children in the wild. How much input data is used to train modern language models?

Do children really lack input data though? Human sensory input is quite a lot of data. Our languages may have common structure because that structure reflects causality and physics encountered by directly sampling the real world.

Keep this as a secret! You are over-scientific and over-intellegent against humanities folks gethered here!

Re: Modern language models refute Chomsky’s approach to language

#100

This paper completly misses the point of linguistics as a discipline that generative grammar operates in. We have known that it is possible to understand language without innateness. That is what linguists do. If you look at how linguists know about innate features, the answer is almost always by first discovering them explicitly while analysing language data; not by opening a brain to see what is innately inside. [0…

Also, dogs and cats, and even our close relatives, primates, don't develop a capacity to language.
Post reply on HN