171–180 of 244 posts
Earlier quoted context omitted.
I didn’t say everyone learns it from scratch at least one, I said we did.
That's not learning, though, is it? Inventing, maybe, or developing or evolving.
Earlier quoted context omitted.
Humans learn from the structure of the world -- not the structure of language. LLMs cheat at generating text because they do so via a model of the statistical structure of text. We're in the world , it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily . What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all lik…
There's nothing qualitatively less "in the world" about a language model than a human. Yes, a human has more senses, and is doubtless exposed to huge categories of training data that a language model doesn't have access to - but it's false to draw a sharp dichotomy between knowing what an iPhone looks like, and knowing how people talk about iPhones. Consider two people - one, a Papau New Guinea tribesperson from a pr…
A concept is a sensory-motor technique abstracted into a pattern of thought developed by an animal, in a spatio-temporal environment, for a purpose.
LLMs are just literally an ensemble of statistical distributions over text symbols. In generating text, they're just sampling from a compressed bank of all text ever digitised.
We aren't sampling from such a bank, we develop wholey non-linguistic concepts which describe the world, and it is these which language piggy-backs on.
The structure of symbols in a book has nothing to do with the structure of the world -- it is we who have stipulated their meaning: there's no meaning to `i`
Earlier quoted context omitted.
>Also, the LLMs are cheating! No...they aren't. Humans aren't learning from thin air by any stretch of the imagination.
Humans invented English, and every other human language. Not one single human, and not in a single lifetime, but yes, we did it. LLMs are not doing this.
Earlier quoted context omitted.
Humans learn from the structure of the world -- not the structure of language. LLMs cheat at generating text because they do so via a model of the statistical structure of text. We're in the world , it is us who stipulate the meaning of words and the structure of text. And we stipulate new meanings to novel parts of the world daily . What else is an 'iPhone' etc. ? There's nothing in `i P h o n e` which is at all lik…
There's nothing qualitatively less "in the world" about a language model than a human. Yes, a human has more senses, and is doubtless exposed to huge categories of training data that a language model doesn't have access to - but it's false to draw a sharp dichotomy between knowing what an iPhone looks like, and knowing how people talk about iPhones. Consider two people - one, a Papau New Guinea tribesperson from a pr…
But then also consider the following: a human being from 2006, and an LLM that has absorbed an enormous corpus of words about iPhones that is also granted access to a capacitive-touchscreen friendly robot arm and continuous feed digital camera (and since I'm feeling generous, also a lot of words about the history and architecture of robot arms and computer vision). There is no doubt the LLM will completely blow the human out of the water if asked trivia questions about the iPhone and its ecosystem.
But my money's on the 2006 human doing a lot better at switching it on and using the Tinder app...
Earlier quoted context omitted.
I mean have you seen the whole "encode a poem in made up emoji" followed by "decode the poem from emoji" stuff? I think it's not unreasonable to think with the right prompts LLMs could do this.
The LLMs that are doing this are trained on trillions of examples of human language. This is not remotely a counterexample. Now if an LLM can invent a new full language from scratch, without any training data of existing languages (like how AlphaZero learned to play Go), then that will be impressive, and a difference in kind.
The author (bafflingly) seems to have completely missed the point- since anything they state up to page 15 (at which point I stopped reading) does not refute Chomsky's points at all. The author talks about LLMs and how they generate text and then goes on to talk about how it refutes Chomsky's claim about syntax and semantics. However it does not since Chomsky's primary claim is about how HUMANS acquire language. The…
We could use programming languages as a counterpoint. LLMs can code in the same way they can use natural languages. But we know that programming languages have structure, we made them that way, from scratch, using Chomsky's theory no less. Saying that because LLMs can learn programming languages using a different approach and therefore disprove the very theory they are built on is absurd. Anyways, the paper is long a…
Very much so, it's astounding really. I still remember deriving "words" and using Chomsky's Normal Form when making the CFG to build a compiler.
Earlier quoted context omitted.
1. No Language model is yet even close to the scale of the human brain 2. Depending on what exactly you're trying to teach (perfect grammar, paragraphs of coherent text, basic reasoning), much less data is needed. https://arxiv.org/abs/2305.07759 3. Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universa…
> Brains don't start at 0. Evolution, dna/rna etc. There's obviously some pre disposition for language learning in humans but that alone isn't enough ground for a "universal grammar" That is literally what universal grammar is. All that is left is to argue about the size and content of UG.
https://www.scientificamerican.com/article/evidence-rebuts-c...
Earlier quoted context omitted.
Chomsky has been adding parameters to his theory to handle exceptions in a way that mimics the endless series of conditional statements appended to knowledge systems of yore. In neuroscience, predictive processing has gained immense favor and can explain language in ways that have nothing to do with innate grammar. https://en.wikipedia.org/wiki/Predictive_coding Exactly how well did "building a bird" work for buildin…
What are these parameters Chomsky has been adding? I'm interested, not to be taken as some defensive remark or the like.
https://en.wikipedia.org/wiki/Generative_grammar
https://en.wikipedia.org/wiki/Transformational_grammar
https://en.wikipedia.org/wiki/X-bar_theory
https://en.wikipedia.org/wiki/Principles_and_parameters
Earlier quoted context omitted.
That's not learning, though, is it? Inventing, maybe, or developing or evolving.
We either have the capacity to make boats and traverse the oceans or we don’t. These days almost nobody makes boats, but we sure used to and still can. What’s your point?
Earlier quoted context omitted.
We did, at least once.
The brain is obviously pre-disposed to language. But evolution fueled learning is by no means from "thin air".
This is a complete tautology in terms, and any dispute must therefore be over the ontology in which it is expressed. To elaborate how something came to be is not required to express that something is.