Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large models of what? Mistaking engineering achievements for linguistic agency

arxiv.org

Post reply on HN
Print view View original

31–40 of 162 posts

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#31

Post by nsagent »

GeneralMayhem wrote: post #10
abernard1 wrote: post #9

Earlier quoted context omitted.

> LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's impossible to gain such an understanding from text. I would argue it is (obviously) impossible the way the current implementation of models work. How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperatu…

The argument would be that that conceptual model is encoded in the intermediate-layer parameters of the model, in a different but analogous way to how it's encoded in the graph and chemical structure of your neurons.

Using Occam's razor, that is less probable than the model picking up on statistical regularities in human language, especially since that's what they are trained to do.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#32

Post by famouswaffles »

abernard1 wrote: post #11
GeneralMayhem wrote: post #10

Earlier quoted context omitted.

The argument would be that that conceptual model is encoded in the intermediate-layer parameters of the model, in a different but analogous way to how it's encoded in the graph and chemical structure of your neurons.

I agree that's an argument. I would contend that argument is obviously false. If it were true, LLMs could multiply scalar numbers together trivially. It should be the easiest thing in the world for them. The network required to do that well is extremely small, the parameter sizes of these models are gigantic, and the textual expression is highly regular: multiplication is the simplest concept imaginable. That they ca…

>If it were true, LLMs could multiply scalar numbers together trivially.

I mean, it's not like GPT-4 can't do this with more accuracy than a human without a calculator.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#33

Post by drdeca »

abernard1 wrote: post #9
GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

> LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's impossible to gain such an understanding from text. I would argue it is (obviously) impossible the way the current implementation of models work. How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperatu…

> I would argue it is (obviously) impossible the way the current implementation of models work.

> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically?

Any probability distribution over strings can theoretically be factored into a product of such a “probability that next token is x given that the text so far is y”. Now, whether a probability distribution over strings can efficiently computed in this form, is another question. But, if we are being so theoretical that we don’t care about the computational cost (as long as it is finite), then the “it is next token prediction” can’t preclude anything which “it produces a probability distribution over strings” doesn’t already preclude.

As for the temperature, given any probability distribution over a discrete set, we can modify it by adding a temperature parameter. Just take the log of the probabilities according to the original probability distribution, scale them all by a factor (the inverse of the temperature, I think. Either that or the temperature, but I think it is the inverse of the temperature.), then exponentiate each of these, and then normalize to produce a probability distribution.

So, the fact that they work by next token prediction, and have a temperature parameter, cannot imply any theoretical limitation that wouldn’t apply to any other way of expressing a probability distribution over strings, as far as discussing probability distributions in the abstract, over strings, rather than talking about computational processes that implement such probability distributions over strings.

But also like, going between P(next token is x | initial string so far is y) and P(the string begins with z) , isn’t that computationally costly? Well, in one direction anyway. Because like, P(next token is x|string so far is y) = P(string begins with yx) / P(string begins with y) .

Though, one might object to P(string starts with y) over P(string is y) ?

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#34

Post by fshbbdssbbgdd »

abernard1 wrote: post #9
GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

> LLMs do not have corporeal experience. But it's not obvious that this means that they cannot, a priori, have an "internal" concept of reality, or that it's impossible to gain such an understanding from text. I would argue it is (obviously) impossible the way the current implementation of models work. How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperatu…

> How could a system which produces a single next word based upon a likelihood and and a parameter called a "temperature" have a conceptual model underpinning it? Even theoretically?

Could a creature that simply evolved to survive and reproduce possibly have a conceptual model underpinning it? Model training and evolution are very different processes, but they are both ways of optimizing a physical system. It may be the case that evolution can give rise to intelligence and model training can’t, but we need some argument to prove that.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#35

Post by kazinator »

The authors of this paper are just another instance of the AI hype being used by people who have no connection to it, to attract some kind of attention.

"Here is what we think about this current hot topic; please read our stuff and cite generously ..."

> Language completeness assumes that a distinct and complete thing such as `a natural language' exists, the essential characteristics of which can be effectively and comprehensively modelled by an LLM

Replace "LLM" by "linguistics". Same thing.

> The assumption of data completeness relies on the belief that a language can be quantified and wholly captured by data.

That's all that a baby has, who becomes a native speaker of their surrounding language. Language acquisition does not imply totality of data. Not every native speaker recognizes exactly the same vocabulary and exactly the same set of grammar rules.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#36

Post by wonnage »

jdietrich wrote: post #17
abernard1 wrote: post #11

Earlier quoted context omitted.

I agree that's an argument. I would contend that argument is obviously false. If it were true, LLMs could multiply scalar numbers together trivially. It should be the easiest thing in the world for them. The network required to do that well is extremely small, the parameter sizes of these models are gigantic, and the textual expression is highly regular: multiplication is the simplest concept imaginable. That they ca…

LLMs do contain conceptual representations and LLMs are capable of abstract reasoning. This is trivially provable by asking them to reason about something that is a) purely abstract and b) not in the training data, e.g. "All floots are gronks. Some gronks are klorps. Are any floots klorps?" Any of the leading LLMs will correctly answer questions of this type much more often than chance.

Or maybe they're just pattern matching on the very particular sentence structure you've chosen. This isn't a convincing example at all

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#38

Post by brookst »

GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

> sufficiently advanced mimicry is not only indistinguishable from the real thing, but at the limit in fact is the real thing

I am continually surprised at how relevant and pervasive one of Kurt Vonnegut’s major insights is: “we are what we pretend to be, so we must be very careful about what we pretend to be”

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#39

Post by brookst »

LetsGetTechnicl wrote: post #20
jdietrich wrote: post #17

Earlier quoted context omitted.

LLMs do contain conceptual representations and LLMs are capable of abstract reasoning. This is trivially provable by asking them to reason about something that is a) purely abstract and b) not in the training data, e.g. "All floots are gronks. Some gronks are klorps. Are any floots klorps?" Any of the leading LLMs will correctly answer questions of this type much more often than chance.

That is not an example of a LLM being capable of abstract reasoning. Changing the question from "What is the capital of United States?" which is easily answerable to something completely abstract and "not in the training model" doesn't change that LLM's are just very advanced text prediction, and always will be. The nature of their design means they are incapable of AGI.

That’s a tautology that seems just as applicable to humans.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#40

Post by hackernewds »

dartos wrote: post #3
mnkv wrote: post #2

Good summary of some of the main "theoretical" criticism of LLMs but I feel that it's a bit dated and ignores the recent trend of iterative post-training, especially with human feedback. Major chatbots are no doubt being iteratively refined on the feedback from users i.e. interaction feedback, RLHF, RLAIF. So ChatGPT could fall within the sort of "enactive" perspective on language and definitely goes beyond the issue…

FWIW even more recently, models have been tuned using a method called DPO instead of RLHF. IIRC DPO doesn’t have human feedback in the loop

DPO most essentially has human feedback, depends on what the preference optimizations are
Post reply on HN