Large models of what? Mistaking engineering achievements for linguistic agency • Hacker News Board

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#131

Post by Terr_ » Sun, Jul 21, 2024, 7:09 PM UTC

GeneralMayhem wrote: post #127

randcraw wrote: post #121

Earlier quoted context omitted.

I think the major lessons from the success of LLMs are two: 1) the astonishing power of a largely trivial association engine based only on the semantic categories inferred by word2vec, and 2) that so much of the communication abilities of the human mind require so little rational thought (since LLMs demonstrate essentially none of the skills in Kahneman's and Tversky's System 2 thinking (logic, circumspection, self-c…

What makes this tough is that LLMs can show logical thinking and self-correction when specifically prompted (e.g. "think step by step", "double-check and then correct your work"). It seems unlikely that they can truthfully self-reflect, but I don't think it's strictly impossible.

> LLMs can show logical thinking and self-correction

The same way they "show" sadness or contrition or excitement?

We need to be careful with our phrasing here: LLMs can be prompted to provide you associated phrases that usually seem to fit with the rest of the word-soup, but whether the model is actually demonstrating "logical thinking" or "self-correction" is a Chinese Room problem [0]. (Or else a "No, it doesn't, I can tell because I checked the code.")

[0] https://en.wikipedia.org/wiki/Chinese_room

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#132

Post by persnickety » Sun, Jul 21, 2024, 7:25 PM UTC

stoperaticless wrote: post #116

persnickety wrote: post #79

Earlier quoted context omitted.

I can't disagree more. Or maybe I actually agree. Because it's not easy to tell whether something is flying. Definitions like that fall apart every time we encounter something out of the ordinary. If you take the criterion of "there's no discussion about it", then you're limiting the definition to that which is familiar, not that which is interesting. Is an ekranoplan flying? Is an orbiting spaceship flying? Is a hov…

> Your criterion would suggest the answer of "no" to any of those cases, even though those cover much of the same use cases as flying, and possibly some new, more interesting ones. Is it a problem though? Their existence are unrelated to how we categorize them. That matters only in communication. “if everybody agrees” lowers/removes the risk of miscommunication. If “hovercraft is flying” for you, but not for 50% the…

I think it matters merely by the way we set our expectations relative to what is going to come - and what has come already. I'm feeling an undercurrent of thought that is implying: this is not X (intelligence, understanding, whatever), so there's no need to consider it seriously.

In the same vein: https://eschwitz.substack.com/p/strange-intelligence-strange...

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#133

Post by dartos » Sun, Jul 21, 2024, 7:54 PM UTC

meroes wrote: post #117

valec wrote: post #4

Earlier quoted context omitted.

it does. that's what the "direct preference" part of DPO means. you just avoid training an explicit reward model on it like in rlhf and instead directly optimize for log probability of preferred vs dispreferred responses

What is it called when humans interact with a model through lengthy exchanges (mostly humans correcting the model’s responses to a posed question to the model, mostly through chat and labeling each statement by the model as correct or not), and then all of that text (possibly with some editing) is fed to another model to train that higher model? Does this have a specific name?

I don’t think that process has a specific name. It’s just how training these models works.

Conversations you have with like chatgpt are likely stored, then sorted through somehow, then added to an ever growing dataset of conversations that would be used to train entirely new models.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#134

Post by plasticeagle » Sun, Jul 21, 2024, 8:18 PM UTC

mitthrowaway2 wrote: post #105

plasticeagle wrote: post #77

Earlier quoted context omitted.

There are many finite problems that absolutely do not admit finite solutions. Full stop. I think the deeper point of the paper is that you simply cannot generate an intelligent entity by just looking at recorded language. You can create a dictionary, and a map - but one must not mistake this map for the territory.

The human brain is a finite solution, so we already have an existence proof. That means a lot for our confidence in the solvability of this kind of problem. It is also not universally impossible to reconstruct a function of finite complexity from only samples of its inputs and outputs. It is sometimes possible to draw a map that is an exact replica of the territory.

Trying to recreate a "human brain" is an absolutely terrible idea - and is not something we should even attempt. The consequences of success are terrible.

They're not really trying to create a human brain, so far as I can tell. They're trying to create an oracle, by feeding it all existing human utterances. This is certainly not going to succeed, since the truth is not measurable post-facto from these utterances.

The claim regarding reconstructing functions from samples of its ins and outs is false. It's false both mathematically, where "finite complexity" doesn't really even have a rigorous definition - and metaphorically too.

Maps are never the territory.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#135

Post by grantcas » Sun, Jul 21, 2024, 8:32 PM UTC

nativeit wrote: post #26

I'm more or less a layperson when it comes to LLMs and this nascent concept of AI, but there's one argument that I keep seeing that I feel like I understand, even without a thorough fluency with the underlying technology. I know that neural nets, and the mechanisms LLMs employ to train and form relational connections, can plausibly be compared to how synapses form signal paths between neurons. I can see how that make…

[dead]

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#136

Post by mitthrowaway2 » Sun, Jul 21, 2024, 9:42 PM UTC

plasticeagle wrote: post #134

mitthrowaway2 wrote: post #105

Earlier quoted context omitted.

The human brain is a finite solution, so we already have an existence proof. That means a lot for our confidence in the solvability of this kind of problem. It is also not universally impossible to reconstruct a function of finite complexity from only samples of its inputs and outputs. It is sometimes possible to draw a map that is an exact replica of the territory.

Trying to recreate a "human brain" is an absolutely terrible idea - and is not something we should even attempt. The consequences of success are terrible. They're not really trying to create a human brain, so far as I can tell. They're trying to create an oracle, by feeding it all existing human utterances. This is certainly not going to succeed, since the truth is not measurable post-facto from these utterances. The…

Sometimes maps are the territory, especially when the territory that is being mapped is itself a map. An accurate map of a map can be a copy of the map that it maps. The human brain's concept of reality is not reality, it's a map of reality. A function trained to predict human outputs can itself contain a map which is arbitrarily similar to the map that a human carries in their own head.

(Finite complexity is rigorously definable, it's just that the definition is domain-specific).

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#137

Post by belter » Sun, Jul 21, 2024, 10:23 PM UTC

GeneralMayhem wrote: post #8

I am highly skeptical of LLMs as a mechanism to achieve AGI, but I also find this paper fairly unconvincing, bordering on tautological. I feel similarly about this as to what I've read of Chalmers - I agree with pretty much all of the conclusions, but I don't feel like the text would convince me of those conclusions if I disagreed; it's more like it's showing me ways of explaining or illustrating what I already belie…

"Beyond the Hype: A Realistic Look at Large Language Models" - https://news.ycombinator.com/item?id=41026484

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#138

Post by Lerc » Sun, Jul 21, 2024, 10:36 PM UTC

imtringued wrote: post #69

Lerc wrote: post #28

Earlier quoted context omitted.

I'm not so sure that view is very widespread amongst people familiar with how LLMs work. Certainly they become more capable with parameters and data, but there are fundamental things that can't be overcome with a basic model and I don't think anyone is seriously arguing otherwise. For instance LLMs are pretty much stateless without their context window. If you treat the raw generated output as the first and final res…

The problem with is that you need the internal monologue to not be subject to training loss, otherwise the internal monologue is restricted to the training distribution. Something people don't seem to grasp is that the training data mostly doesn't contain any reasoning. Nobody has published brain activity recordings on the internet, only text written in human language. People see information, process it internally in…

I think you are looking at a too narrowly defined avenue to achieve this effect.

There are multiple avenues to train a model to do this. Most simply is a finetune on training examples where the the internal monologue is constructed in a manner that precedes the tag and provides additional reasoning before the output.

I think there is also scope for pretraining with a mask to not attempt to predict (or ignore the loss, same thing) certain things in the stream. For example to give time codes into the data stream. The training could then have an awareness of the passing of time but would not generate time codes as a prediction. Time codes could then be injected into the context at inference time and it would be able to use that data.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#139

Post by stoperaticless » Sun, Jul 21, 2024, 10:48 PM UTC

persnickety wrote: post #132

stoperaticless wrote: post #116

Earlier quoted context omitted.

> Your criterion would suggest the answer of "no" to any of those cases, even though those cover much of the same use cases as flying, and possibly some new, more interesting ones. Is it a problem though? Their existence are unrelated to how we categorize them. That matters only in communication. “if everybody agrees” lowers/removes the risk of miscommunication. If “hovercraft is flying” for you, but not for 50% the…

I think it matters merely by the way we set our expectations relative to what is going to come - and what has come already. I'm feeling an undercurrent of thought that is implying: this is not X (intelligence, understanding, whatever), so there's no need to consider it seriously. In the same vein: https://eschwitz.substack.com/p/strange-intelligence-strange...

> I'm feeling an undercurrent of thought that is implying: this is not X, so there's no need to consider it seriously.

True. I doubt that field experts are directly affected by the naming, but indirect effect might come via less knowledgable (AI wise) financial decision makers.

I see a risk that those decision makers (and society) would be mislead if they were promised AGI (based on their “strict” understanding, what’s in the movies), but received AGI (based on “relaxed” meaning). Informed consent is usually good.

Though surely that can be resolved with more public discourse; maybe “relaxed” version will become the default expectation.

Re: Large models of what? Mistaking engineering achievements for linguistic agency

#140

Post by famouswaffles » Sun, Jul 21, 2024, 11:20 PM UTC

YeGoblynQueenne wrote: post #100

persnickety wrote: post #79

Earlier quoted context omitted.

I can't disagree more. Or maybe I actually agree. Because it's not easy to tell whether something is flying. Definitions like that fall apart every time we encounter something out of the ordinary. If you take the criterion of "there's no discussion about it", then you're limiting the definition to that which is familiar, not that which is interesting. Is an ekranoplan flying? Is an orbiting spaceship flying? Is a hov…

There are going to be gray areas of course, but the point I'm making is that if it's hard to argue something isn't flying (respectively, intelligent) then it's probably flying (resp. intelligent). If it's hard to tell then it's probably not. I'm suggesting that intelligence, like flying, should be very immediately obvious. For example, you can't miss the fact that a five-year old child is intelligent and you can't mi…

>but when something is intelligent then it should leave us no doubt that it is.

Not that long ago, a whole lot of humans (the majority in some continents) asserted other a group of other humans were not intelligent so strongly they purchased them as property and treated them worse than even farm animals, so i think you can basically throw this one out the window.