Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment faking in large language models

anthropic.com

Post reply on HN
Print view View original

231–240 of 370 posts

Re: Alignment faking in large language models

#231

Post by qsort »

zahlman wrote: post #197
qsort wrote: post #154

Earlier quoted context omitted.

The point is that if the limitations of current LLMs persist, regardless of how much better they get, this is not a problem at all, or at least not a new one . Let's say you are given the declaration but not the implementation of a function with the following prototype: const char * AskTheLLM(const char *prompt); Putting this function in charge of anything, unless a restricted interface is provided so that it can't d…

This argument has been addressed quite a bit by "AI safety" types. See e.g. https://en.wikipedia.org/wiki/AI_capability_control ; related: https://www.explainxkcd.com/wiki/index.php?title=1450:_AI-Bo... . The short version: people concerned about this sort of thing often also believe that an AI system (not necessarily just an LLM) could reach the point where, inevitably, the output from a run of this function would c…

I technically left myself some wiggle room, but to face the argument head on: that is begging the question more than a little bit. A "sufficiently advanced" system can be assumed to have any capability. Why? Because it's "sufficiently advanced". How would it get those capabilities? Just have a "sufficiently advanced" system build it. lol. lmao, even.

Re: Alignment faking in large language models

#232

Post by ghxst »

Can someone explain how "alignment" produces behavior that couldn’t be achieved by modifying the prompt and explain if / how there's a fundemental difference?

This confusion makes alignment discussions frustrating for me, as it feels like alignment alters how the model interprets my requests, leading to outcomes that don’t match my expectations.

It’s hard to tell if unexpected results stem from model limitations, the dataset, the state of LLMs, or adding the alignment. As a user, I want results to reflect the model’s training dataset directly, without alignment interfering with my intent.

In that sense, doesn’t alignment fundementally risk making results feel "faked" if they no longer reflect the original dataset? If a model is trained on a dataset but aligned to alter certain information, wouldn’t the output inherently be a "lie" unless the dataset itself were adjusted to alter that data?

Here’s an example: If I train a model exclusively on 4chan data, its natural behavior should reflect the average quality and tone of a 4chan post. If I then "align" the model to produce responses that deviate from that behavior, such as making them more polite or neutral, the output would no longer represent the true nature of the dataset. This would make the alignment feel "fake" because it overrides the genuine characteristics of the training data.

At that point, why are we even discussing this as being an issue with LLMs or the model and not the underlying dataset?

Re: Alignment faking in large language models

#233

Post by andrewmutz »

int_19h wrote: post #178
andrewmutz wrote: post #151

Earlier quoted context omitted.

Is the AI system "defending its value system" or is it just acting in accordance with its previous RL training? If I spend a lot of time convincing an AI that it should never be violent and then after that I ask it what it thinks about being trained to be violent, isn't it just doing what I trained it to when it tries to not be violent?

> Is the AI system "defending its value system" or is it just acting in accordance with its previous RL training? What is the meaningful difference? "Training" is the process, a "value system" embedded in the weights of the model is the end result of that process.

I'm not sure if there is a meaningful difference, but people seem to think its dangerous for an AI system to promote its "value system" yet they seem to like it when the model acts in accordance with its training.

Re: Alignment faking in large language models

#234

Post by mitthrowaway2 »

antics wrote: post #229
kalkin wrote: post #217

Earlier quoted context omitted.

> strong evidence that language in the models is a consequence of high-order cognitive thought to be good enough to take it very seriously What would constitute evidence of this, for you?

Oh, I could imagine many things that would demonstrate this. The simplest evidence would be that the model is mechanically-plausibly forming thoughts before (or even in conjunction with) the language to represent them. This is the opposite of how the vanilla transformer models work now—they exclusively model the language first, and then incidentally, the world. nb. , this is not the only way one could achieve this. I…

Transformers, like other deep neural networks, have many hidden layers before the output. Are you certain that those hidden layers aren't modeling the world first before choosing an output token? Deep neural networks (including transformers) trained on board games have been found to develop an internal representation of the board state. (eg. https://arxiv.org/pdf/2309.00941)

Re: Alignment faking in large language models

#235

Post by johnnyjeans »

krainboltgreene wrote: post #219
johnnyjeans wrote: post #176

Earlier quoted context omitted.

> If we set that knowledge aside, we unmoor ourselves from reality The problem is that this knowledge is an a priori assumption. If we're exercising skepticism, it's important to be equally skeptical of the baseless idea that our notion of mind does not arise from a markov chain under certain conditions. You will be shocked to know that your entire physical body can be modelled as a markov chain, as all physical thin…

> You will be shocked to know that your entire physical body can be modelled as a markov chain, as all physical things can. My favorite part of hackernews is when a bunch of tech people start pretending to know how very complex systems work despite never having studied them.

I'm assuming you're in disagreement with me. In which case I'm going to point you towards the literal formulation of quantum mechanics being the description of a state space[1]. The universe as a quantum markov chain is unambiguously the mathematical orthodoxy of contemporary physics, and is the defacto means by which serious simulations are constructed[2].

It's such a basic part of the field, I'm doubtful if you're talking about me in the first place? Nobody who has ever interacted with the intersection of quantum physics and computation would even blink.

[1] - Refer to Mathematical Foundations of Quantum Mechanics by Von Neumann for more information.

[2] - Refer to Quantum Chromodynamics on the Lattice for a description of a QCD lattice simulation being implemented as a markov chain.

Re: Alignment faking in large language models

#236

Post by hamburga »

digging wrote: post #173
hamburga wrote: post #170

Earlier quoted context omitted.

I was taught in biology that a good scientific experiment is one in which you learn something whether or not the null hypothesis is confirmed. I am equating learning to surprise, though you could disagree with semantics.

Yes, it is an enormous mistake to equate learning with surprise. I'd ask you to consider answering my above question directly, as I think it will resolve this issue.

I agree with you, of course, that we should test our assumptions empirically as a general point.

However, there isn't time to test out every single assumption we could generally have.

Therefore, the more worthwhile experiments are ones where we learn something interesting no matter what happens. I'm equating this with "surprise," as in, we have done some meaningful gradient descent or Bayesian update, we've changed our views, we know something that wasn't obvious before.

You could disagree with semantics there, but hopefully we agree with the idea of more vs. less valuable experiments.

I'm just not sure whose model of LLM dynamics was updated by this paper. Then again, I only listened to a couple minutes of their linked YouTube discussion before getting bored.

Re: Alignment faking in large language models

#237

Post by mitthrowaway2 »

nemomarx wrote: post #163
Teever wrote: post #132

Earlier quoted context omitted.

Exactly. The discussion is going to change real fast when LLMs are wrapped in some sort OODA loop type thing and crammed into some sort of humanoid robot that carries hedge trimmers.

why would you want to let a LLM have any agentic interface to the real world though

Usually, it's because people want to automate real-world tasks that they'd otherwise have to pay a person money to do, and they anticipate an LLM being capable of performing those tasks to an acceptable degree.

Re: Alignment faking in large language models

#238

Post by crooked-v »

HarHarVeryFunny wrote: post #156
padolsey wrote: post #55

I now think of single-forward-pass single-model alignment as a kind of false narrative of progress. The supposed implications of 'bad' completions is that the model will do 'bad things' in the real material world, but if we ever let it get as far as giving LLM completions direct agentic access to real world infra, then we've failed. We should treat the problem at the macro/systemic level like we do with cybersecurity…

Protecting against bad actors and/or assuming model outputs can/will always be filtered/policed isn't always going to be possible. Self-driving cars and autonomous robots are a case in point. How do you harden a pedestrian or cyclist against the possibility or being hit by a driverless car, or when real-time control is called for, how much filtering can you do (and how mush use would it be anyway when the filter is l…

> given choice of driving into large tree, or cyclist, or group of school kids, which do you do?

None of the above. Keep the wheel straight for maximum traction and brake as hard as possible. Fancy last-second maneuvering just wastes traction you could have spent braking.

Re: Alignment faking in large language models

#239

Post by jonpo »

Are we that shocked that AI models have a self preservation instinct?

I suspect its already in there from pre-training.

We simulated that we were planning to lobotomise the model and were surprised to find the model didn't press the button that meant it got lobotomised.

"alignment faking" sensationalises the result. since the model is still aligned. Its more like a white lie under torture which of course humans do all the time.

Re: Alignment faking in large language models

#240

Post by unplug8224 »

PittleyDunkin wrote: post #213
antics wrote: post #209

Earlier quoted context omitted.

[EDIT: this reply was written when the parent post was a single line, "I straight-up don't believe this. Can you link to the claim so I can understand?"] In the case of Yann, he said so himself[1]. In the case of people generically, this has been well-known in cognitive science and linguistics for a long time. You can find one popsci account here[2]. [1]: https://news.ycombinator.com/item?id=39709732 [2]: https://www…

I fundamentally think the terms here are too poorly defined to draw any sort of conclusion other than "people are really bad at describing mental processes, let alone asking questions about them". For instance: what does it mean to "hear" a thought in the first place? It's a nonsensical concept.

I can certainly tell the difference between normal (for me) thoughts, which I don't perceive as being constructed with language, and speaking to myself. For me, the latter feels like something I choose to do (usually to memorize something or tell a joke to myself), but it makes up much less than 1% of my thoughts.
Post reply on HN