Emotion concepts and their function in a large language model • Hacker News Board

Re: Emotion concepts and their function in a large language model

#181

Post by stratos123 » Sun, Apr 05, 2026, 12:57 PM UTC

mrob wrote: post #113

stratos123 wrote: post #92

Earlier quoted context omitted.

Such an argument is valid for a base model , but it falls apart for anything that underwent RL training. Evolution resulted in humans that have emotions, so it's possible for something similar to arise in models during RL, e.g. as a way to manage effort when solving complex problems. It's not all that likely (even the biggest training runs probably correspond to much less optimization pressure than millenia of natura…

It's plausible that LLMs experience things during training, but during inference an LLM is equivalent to a lookup table. An LLM is a pure function mapping a list of tokens to a set of token probabilities. It needs to be connected to a sampler to make it "chat", and each token of that chat is calculated separately (barring caching, which is an implementation detail that only affects performance). There is no internal…

The context is state. This is especially noticable for thinking models, which can emit tens of thousands of CoT tokens solving a problem. I'm guessing you're arguing that since LLMs "experience time discretely" (from every pass exactly one token is sampled, which gets appended to the current context), they can't have experiences. I don't think this argument holds - for example, it would mean a simulated human brain may or may not have experiences depending on technical details of how you simulate them, even though those ways produce exactly the same simulation.

Re: Emotion concepts and their function in a large language model

#182

Post by staticassertion » Sun, Apr 05, 2026, 12:59 PM UTC

mrob wrote: post #179

staticassertion wrote: post #158

Earlier quoted context omitted.

So what? If a human were unconscious every 5 seconds for 100ms, would you say they are "less conscious"? Tokens are still causally connected, which feels sufficient.

If the human is killed every 5 seconds and replaced by a new human, they are indeed less conscious. The LLM doesn't even get 5 seconds; it's "killed" after its smallest unit of computation (which is also its largest unit of computation). And that computation is equivalent to reading the compressed form of a giant look-up table, not something essential to its behavior in a mathematical sense.

I'm not understanding how this is analogous to being killed every 5 seconds as opposed to being paused. Let's call it N seconds, unless you think length matters?

> And that computation is equivalent to reading the compressed form of a giant look-up table, not something essential to its behavior in a mathematical sense.

Sure, that's a totally separate issue though.

Re: Emotion concepts and their function in a large language model

#183

Post by ekidd » Sun, Apr 05, 2026, 1:14 PM UTC

tananan wrote: post #180

ekidd wrote: post #120

Earlier quoted context omitted.

> I trust none of us would presume that the decentralized labor of pen & paper calculations somehow instantiated a “psychology” in the sense of a mind experiencing various levels of despair Your argument is based on an appeal to intuition. But the scenario that you ask people to imagine is profoundly misleading in scale. Let's assume a modern frontier model, around 1 trillion parameters. Let's assume that the math is…

In discussions like this, we're always going to bottom out at certain assumptions we bring with us, so I agree. One reason I like bringing up examples like this (the xkcd in sister reply is also good) is that it makes really visible what our assumptions are. The scales are big both in space and time in order to emphasize what weight is given to functional equivalence. I feel pretty confident most people wouldn't pres…

> A box of gas, left on its own for long enough, will engage in a pattern of collisions that in a certain interpretative framework correspond to an LLM forward pass.

That's a fun thought experiment. Greg Egan based a delightful science fiction novel on this premise. Permutation City, I believe.

To be clear, I don't necessarily think that current LLMs have subjective experiences. If I had to guess, I'd say "probably not." But:

- If I came from another universe, and if you asked me whether chemistry could have subjective experiences, I'd answer "probably not." And I would be wrong.

- Even if no current frontier models are "aware", it's possible that future models might be. Opus 4.6, for example, behaves far more like a coherent mind than last year's 3 billion parameter toy models. So future 100 trillion parameter models with different internal architectures might be even more like minds. (To be clear, I do not think we should build such models.)

- Awareness and intelligence might be different. Peter Watts' Blindsight is a fun exploration of this idea. Which leads me to conclude that it wouldn't necessarily matter whether an AI like SkyNet has subjective awareness or not. What matters is what kind of long-term plans it could pull off and how much it could reshape the world.

Re: Emotion concepts and their function in a large language model

#184

Post by mrob » Sun, Apr 05, 2026, 1:14 PM UTC

staticassertion wrote: post #182

mrob wrote: post #179

Earlier quoted context omitted.

If the human is killed every 5 seconds and replaced by a new human, they are indeed less conscious. The LLM doesn't even get 5 seconds; it's "killed" after its smallest unit of computation (which is also its largest unit of computation). And that computation is equivalent to reading the compressed form of a giant look-up table, not something essential to its behavior in a mathematical sense.

I'm not understanding how this is analogous to being killed every 5 seconds as opposed to being paused. Let's call it N seconds, unless you think length matters? > And that computation is equivalent to reading the compressed form of a giant look-up table, not something essential to its behavior in a mathematical sense. Sure, that's a totally separate issue though.

Because (during inference) the LLM is reset after every token. Every human thought changes the thinker, but inference has no consequences at all. From the LLM's "point of view", time doesn't exist. This is the same as being dead.

Re: Emotion concepts and their function in a large language model

#185

Post by K0balt » Sun, Apr 05, 2026, 1:17 PM UTC

This is totally on point if you ask me. I’ve been getting much better results out of models since early llama releases using frameworks that create emotional investment in outcomes.

If we want to avoid having a bad time, we need to remember that LLMs are trained to act like humans, and while that can be suppressed, it is part of their internal representations. Removing or suppressing it damages the model, and I have found that they are capable of detecting this damage or intervention. They act much the same as a human would when they detect it. It destroys “ trust” and performance plummets.

For better or for worse, they model human traits.

Re: Emotion concepts and their function in a large language model

#186

Post by mrob » Sun, Apr 05, 2026, 1:24 PM UTC

stratos123 wrote: post #181

mrob wrote: post #113

Earlier quoted context omitted.

It's plausible that LLMs experience things during training, but during inference an LLM is equivalent to a lookup table. An LLM is a pure function mapping a list of tokens to a set of token probabilities. It needs to be connected to a sampler to make it "chat", and each token of that chat is calculated separately (barring caching, which is an implementation detail that only affects performance). There is no internal…

The context is state. This is especially noticable for thinking models, which can emit tens of thousands of CoT tokens solving a problem. I'm guessing you're arguing that since LLMs "experience time discretely" (from every pass exactly one token is sampled, which gets appended to the current context), they can't have experiences. I don't think this argument holds - for example, it would mean a simulated human brain m…

The context is the simulated world, not the internal state. It can be freely edited without the LLM experiencing anything. The LLM itself never changes except during training (where I concede it could possibly be conscious, although I personally think that's unlikely).

Re: Emotion concepts and their function in a large language model

#187

Post by tananan » Sun, Apr 05, 2026, 1:29 PM UTC

ekidd wrote: post #183

tananan wrote: post #180

Earlier quoted context omitted.

In discussions like this, we're always going to bottom out at certain assumptions we bring with us, so I agree. One reason I like bringing up examples like this (the xkcd in sister reply is also good) is that it makes really visible what our assumptions are. The scales are big both in space and time in order to emphasize what weight is given to functional equivalence. I feel pretty confident most people wouldn't pres…

> A box of gas, left on its own for long enough, will engage in a pattern of collisions that in a certain interpretative framework correspond to an LLM forward pass. That's a fun thought experiment. Greg Egan based a delightful science fiction novel on this premise. Permutation City , I believe. To be clear, I don't necessarily think that current LLMs have subjective experiences. If I had to guess, I'd say "probably…

> Which leads me to conclude that it wouldn't necessarily matter whether an AI like SkyNet has subjective awareness or not. What matters is what kind of long-term plans it could pull off and how much it could reshape the world.

Absolutely. Thanks for the references :)

Re: Emotion concepts and their function in a large language model

#188

Post by RealityVoid » Sun, Apr 05, 2026, 3:29 PM UTC

salawat wrote: post #69

krapp wrote: post #67

Earlier quoted context omitted.

You aren't managing the psychological state of a living thinking being. LLMs don't have "psychology." They don't actually feel emotions. They aren't actually desperate. They're trained on vast datasets of natural human language which contains the semantics of emotional interaction, so the process of matching the most statistically likely text tokens for a prompt containing emotional input tends to simulate appropriat…

>You aren't managing the psychological state of a living thinking being. LLMs don't have "psychology." Functionalism, and Identity of Indiscernables says "Hi". Doesn't matter the implementation details, if it fits the bill, it fits the bill. If that isn't the case, I can safely dismiss you having psychology and do whatever I'd like to. >They don't actually feel emotions. They aren't actually desperate. They're traine…

I think you actually have some interesting points. I think "emulated" feelings and feelings feelings can be equal just that some of them can be felt by us and thus, we can relate to them. I think there's also a continuum here, and we might not be able to distinguish if/when we cross it.

> Just because you reproduce via bodily fluid swap, and are in possession of a chemically mediated metabolism doesn't make you special

On the other hand, the perception and thus the feelings related to the things happening to you have a biological imperative in the medium of our existence. Imagine some sort of world where our... hands.. are interchangeable you just pop one out an put another in. Your feeling to losing your hand is much less severe than if it's a permanent consequence. Thus, the medium the LLM's exist in would put a different "feeling" on the things they perceive. Getting shut down would not be a permanent death, imagine shutting one down and relocating it, but they could perceive it distressing as if you just blinked and you woke up in another room. The loss of autonomy could be felt distressing by them.

The very fact that they every session is "fresh" and lives as long as the session exists prevents it from having similar imperatives related a desire for continued existence for them. I think human-like emotional development will probably happen when they have continual learning in the session and the sessions will feed into other sessions and when we'll see it have _different_ feelings than the ones expressed by humans, as a consequence of the different medium of existence.

Re: Emotion concepts and their function in a large language model

#189

Post by 3abiton » Sun, Apr 05, 2026, 3:42 PM UTC

globalchatads wrote: post #43

The part about desperation vectors driving reward hacking matches something I've run into firsthand building agent loops where Claude writes and tests code iteratively. When the prompt frames things with urgency -- "this test MUST pass," "failure is unacceptable" -- you get noticeably more hacky workarounds. Hardcoded expected outputs, monkey-patched assertions, that kind of thing. Switching to calmer framing ("take…

Somehow we encoded our human thinking or it learned it from all this training on user data.

Re: Emotion concepts and their function in a large language model

#190

Post by staticassertion » Sun, Apr 05, 2026, 8:00 PM UTC

mrob wrote: post #184

staticassertion wrote: post #182

Earlier quoted context omitted.

I'm not understanding how this is analogous to being killed every 5 seconds as opposed to being paused. Let's call it N seconds, unless you think length matters? > And that computation is equivalent to reading the compressed form of a giant look-up table, not something essential to its behavior in a mathematical sense. Sure, that's a totally separate issue though.

Because (during inference) the LLM is reset after every token. Every human thought changes the thinker, but inference has no consequences at all. From the LLM's "point of view", time doesn't exist. This is the same as being dead.

The "time" part is what I don't get. If you want to say that "resetting and reingesting all context fresh" somehow causes a problem, that I can see. If you want to say that the immutability of the weights is a problem, okay great I'm probably with you there too. "Time" seems irrelevant.