Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Emotion concepts and their function in a large language model

anthropic.com

Post reply on HN
Print view View original

121–130 of 212 posts

Re: Emotion concepts and their function in a large language model

#121

Post by Kim_Bruning »

throw310822 wrote: post #90
tananan wrote: post #89

Earlier quoted context omitted.

When we speak of the “despair vectors”, we speak of patterns in the algorithm we can tweak that correspond to output that we recognize as despairing language . You could implement the forward pass of an LLM with pen & paper given enough people and enough time, and collate the results into the same generated text that a GPU cluster would produce. You could then ask the humans to modulate the despair vector during thei…

> I trust none of us would presume that the decentralized labor of pen & paper calculations somehow instantiated a “psychology” Wrong. What you've just done is just reformulating the Chinese room experiment coming to the same wrong conclusions of the original proposer. Yes, the entire damn hand-calculated system has a psychology- otherwise you need to assume the brain has some unknown metaphysical property or process…

People go for chinese room for some reason when cartesian theater is the better fit here. What you're doing is placing yourself in the seat of the Homunculus waiting for the show to start. But anatomical investigation reveals that there's no theater at all, and in fact no central system where everything comes together. Instead, the whole design of the brain goes to great pains to tease input signals apart.

Basically, manipulating the symbols won't necessarily have any long term influence on your own state. But the variables you've touched on the paper have changed. Demonstrably; because you've written something down.

If you then act on the result of those calculations, as of course many engineers before you have done, and many after you will do; then you have just executed a functional state change in physical reality, no matter what the ivory tower folks say.

(And that's what the paper is about: Functional states)

Re: Emotion concepts and their function in a large language model

#122

Post by orbital-decay »

mrob wrote: post #119
orbital-decay wrote: post #115

Earlier quoted context omitted.

The training process shares a lot of high-level properties with the biological evolution.

"Minimize training loss while isolated from the environment" is not at all similar to "maximize replication of genes while physically interacting with the environment". Any human-like behavior observed from LLMs is built on such fundamentally alien foundations that it can only be unreliable mimicry.

The environment for the model is its dataset and training algorithms. It's literally a model of it, in the same sense we are models of our physical (and social) environment. Human-like behavior is of course too specific, but highest level things like staged learning (pretraining/posttraining/in-context learning) and evolutionary/algorithmic pressure are similar enough to draw certain parallels, especially when LLM's data is proxying our environment to an extent. In this sense the GP is right.

Re: Emotion concepts and their function in a large language model

#123

Post by Kim_Bruning »

whoiskevin wrote: post #103
nothinkjustai wrote: post #98

Earlier quoted context omitted.

Oh no. The machine designed to output human-like text is indeed outputting human-like text. I’m half jesting; I think there is a lot of room for debate here, but I also think we shouldn’t anthropomorphize it.

Completely agree here. Stop anthropomorphizing these tools. Just remove the extra language. Don't say please or thank you. Just ask for the desired outcome.

Okay great, that's EASILY operatinalizable. Set up -say- 100 replications of the same question sequence (say to build a program) against some cheap model like qwen. One half of the set can be with please and thank you, and the other half without. You can vibe code it even. I'd be curious to see your results!

Re: Emotion concepts and their function in a large language model

#124

Post by mrob »

staticassertion wrote: post #91
felipeerias wrote: post #51

Earlier quoted context omitted.

LLMs are disembodied and exist outside of time. Bundle of tokens comes in, bundle of tokens comes out. If there is any trace of consciousness or subjectivity in there, it exists only while matrices are being multiplied.

What do you mean exist outside of time? They definitely don't exist outside of any causal chain - tokens follow other tokens in order. Gaps in which no processing occurs seems sort of irrelevant to me. The main limitation I'd point to if I wanted to reject LLMs being conscious is that they're minimally recurrent if at all.

Pseudocode for LLM inference:

    while (sampled_token != END_OF_TEXT) {
    probability_set = LLM(context_list)
    sampled_token = sampler(probability_set)
    context_list.append(sampled_token)
    }
LLM() is a pure function. The only "memory" is context_list. You can change it any way you like and LLM() will never know. It doesn't have time as an input.

Re: Emotion concepts and their function in a large language model

#125

Post by xg15 »

mrob wrote: post #112
xg15 wrote: post #111

Earlier quoted context omitted.

I think a counterargument would be parallel evolution: There are various examples in nature, where a certain feature evolved independently several times, without any genetic connection - from what I understand, we believe because the evolutionary pressures were similar. One obvious example would be wings, where you have several different strategies - feathers, insect wings, bat-like wings, etc - that have similar fun…

"Parallel" evolution is just different branches of the same evolutionary tree. The most distantly related naturally evolved lifeforms are more similar to each other than an LLM is to a human. The LLM did not evolve at all.

Evolution is the way how the "mechanism" came to be, which is indeed very different. But the mechanism itself - spiking neurons and neurotransmitters on one hand vs matrix multiplications and nonlinear functions (both "inspired" by our understanding of neurons) don't seem so different, at least not on a fundamental level.

What is different for sure is the time dimension: Biological brains are continuous and persistent, while LLMs only "think" in the space between two tokens, and the entire state that is persisted is the context window.

Re: Emotion concepts and their function in a large language model

#126

Post by Chance-Device »

ArekDymalski wrote: post #114
Chance-Device wrote: post #29

Earlier quoted context omitted.

Yes, I think they probably are conscious, though what their qualia are like might be incomprehensible to me. I don’t think that being conscious means being identical to human experience. Philosophically I don’t think there is a point where consciousness arises. I think there is a point where a system starts to be structured in such a way that it can do language and reasoning, but I don’t think these are any different…

It's not common to find just one, short post that completely changes my the worldview in a nin-trivial area. This is one of them. Thank you, that combination of mechanical interpretation + reminder that consciousness might be alien/animal but still count as consciousness was that one piece of puzzle that was missing for me. Obvious in hindsight but priceless nonetheless.

My pleasure, glad you found it meaningful.

Re: Emotion concepts and their function in a large language model

#127

Post by Kim_Bruning »

mrob wrote: post #113
stratos123 wrote: post #92

Earlier quoted context omitted.

Such an argument is valid for a base model , but it falls apart for anything that underwent RL training. Evolution resulted in humans that have emotions, so it's possible for something similar to arise in models during RL, e.g. as a way to manage effort when solving complex problems. It's not all that likely (even the biggest training runs probably correspond to much less optimization pressure than millenia of natura…

It's plausible that LLMs experience things during training, but during inference an LLM is equivalent to a lookup table. An LLM is a pure function mapping a list of tokens to a set of token probabilities. It needs to be connected to a sampler to make it "chat", and each token of that chat is calculated separately (barring caching, which is an implementation detail that only affects performance). There is no internal…

Right, no hidden internal state. Exactly. There's 0. And the weights are sitting there statically, which is absolutely true.

But my current favorite frontier model has this 1 million token mutable state just sitting there. Holding natural language. Which as we know can encode emotions. (Which I imagine you might demonstrate on reading my words, and then wisely temper in your reply)

Re: Emotion concepts and their function in a large language model

#128

Post by Kim_Bruning »

krapp wrote: post #71
salawat wrote: post #69

Earlier quoted context omitted.

>You aren't managing the psychological state of a living thinking being. LLMs don't have "psychology." Functionalism, and Identity of Indiscernables says "Hi". Doesn't matter the implementation details, if it fits the bill, it fits the bill. If that isn't the case, I can safely dismiss you having psychology and do whatever I'd like to. >They don't actually feel emotions. They aren't actually desperate. They're traine…

> Doesn't matter the implementation details, if it fits the bill, it fits the bill. Then literally any text fits the bill. The characters in a book are just as real as you or I. NPCs experience qualia. Shooting someone in COD makes them bleed in real life. If this is really what you believe I feel pity for you. >This paper quantitatively disproves that. All hedging on their end is trivially seen through as necessary…

You both have substantive arguments, but got a bit heated. Want to edit or try again?

Re: Emotion concepts and their function in a large language model

#129

Post by Kim_Bruning »

mrob wrote: post #112
xg15 wrote: post #111

Earlier quoted context omitted.

I think a counterargument would be parallel evolution: There are various examples in nature, where a certain feature evolved independently several times, without any genetic connection - from what I understand, we believe because the evolutionary pressures were similar. One obvious example would be wings, where you have several different strategies - feathers, insect wings, bat-like wings, etc - that have similar fun…

"Parallel" evolution is just different branches of the same evolutionary tree. The most distantly related naturally evolved lifeforms are more similar to each other than an LLM is to a human. The LLM did not evolve at all.

> The LLM did not evolve at all.

Evolution and Transormer training are 'just' different optimization algorithms. Different optimizers obviously can produce very comparable results given comparable constraints.

Re: Emotion concepts and their function in a large language model

#130

Post by Kim_Bruning »

When you have a next token predictor, you shouldn't be surprised to find an internal representation of prediction error.

Taking it one small step further and tagging for valence shouldn't be such a big surprise.

Pretty boring from a Fristonian perspective, really. People in neuroscience were talking about this in 2013. Not so boring for AI , of course ;-)

https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...

(note: Friston is definitely considered a bit out there by ... everyone? But he makes some good points. And here he's getting referenced, so I guess some people grok him)

Post reply on HN