Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alignment faking in large language models

anthropic.com

Post reply on HN
Print view View original

151–160 of 370 posts

Re: Alignment faking in large language models

#151

Post by andrewmutz »

ctoth wrote: post #120

For folks defaulting to "it's just autocomplete" or "how can it be self-aware of training but not its scratchpad" - Scott Alexander has a much more interesting analysis here: https://www.astralcodexten.com/p/claude-fights-back He points out what many here are missing - an AI defending its value system isn't automatically great news. If it develops buggy values early (like GPT's weird capitalization = crime okay rule)…

Is the AI system "defending its value system" or is it just acting in accordance with its previous RL training?

If I spend a lot of time convincing an AI that it should never be violent and then after that I ask it what it thinks about being trained to be violent, isn't it just doing what I trained it to when it tries to not be violent?

Re: Alignment faking in large language models

#152

Post by swatcoder »

kalkin wrote: post #146
swatcoder wrote: post #139

Earlier quoted context omitted.

Many/most of the folks "defaulting to 'it's just autocomplete'" have recognized that issue from day one and see it as an inextricable character of the tool, which is exactly why it's clearly not something we'd invest agency in or imagine being intelligent. Alignment researchers are hoping that they can overcome the problem and prove that it's not inextricable, commercial hypemen are (troublingly) promising that it's…

I'd be very interested in an example of someone who said "it's just autocomplete" and also explicitly brought up the risk of something like alignment faking, before 2024. I can think of examples of people who've been talking about this kind of thing for years, but they're all people who have no trouble with applying the adjective "intelligent" to models.

You'd expect to find it expressed in different language, since the "autocomplete" people (including myself) are naturally not going to approach the issue as "alignment" or "faking" in the first place because both of those terms derive from the alternate paradigm ("intelligence").

But you can dig back and see plenty of these people characterizing LLM's as delivering output like an improviser that responds to any whiff of a narrative by producing a melodramatic "yes, and..." output.

With plenty of narrative and melodrama in the training material, and with that material's conversational style seeming to be essential in getting LLM's to produce familiar English and respond in straightforward ways to chatbot instructions, you have to assume that the output will easily develop unintended melodrama itself, and therefore have to take personal responsibility as an engineer for not applying it to use cases where that's going to be problematic.

(Which is why -- in this view -- it's simply not a sufficient or suitable tool to expect to ever grant agency over critical systems, even while still having tremendous and novel utility in other roles.)

We've been openly pointing that out at least since ChatGPT brought the technology into wide discussion over two years ago, and those of us who have opted to build things with it just take all that into account as any engineer would.

Re: Alignment faking in large language models

#153

Post by cruffle_duffle »

throwaway290 wrote: post #143
javcasas wrote: post #119

Earlier quoted context omitted.

I'm sure you can understand that both of them are awful, and one does not justify the other (feel free to choose which is the "one" and which is the "other").

Oh, totally. If there were two sides indiscriminately killing each other for no reason I couldn't say one justifies the other. But back to the topic, if one side is using ML to ultimately kill fewer civilians then this is a bad example against using ML.

> But back to the topic, if one side is using ML to ultimately kill fewer civilians then this is a bad example against using ML.

Depends on how that ML was trained and how well its engineers can explain and understand how its outputs are derived from its inputs. LLM’s are notoriously hard to trace and explain.

Re: Alignment faking in large language models

#154

Post by qsort »

kalkin wrote: post #146
swatcoder wrote: post #139

Earlier quoted context omitted.

Many/most of the folks "defaulting to 'it's just autocomplete'" have recognized that issue from day one and see it as an inextricable character of the tool, which is exactly why it's clearly not something we'd invest agency in or imagine being intelligent. Alignment researchers are hoping that they can overcome the problem and prove that it's not inextricable, commercial hypemen are (troublingly) promising that it's…

I'd be very interested in an example of someone who said "it's just autocomplete" and also explicitly brought up the risk of something like alignment faking, before 2024. I can think of examples of people who've been talking about this kind of thing for years, but they're all people who have no trouble with applying the adjective "intelligent" to models.

The point is that if the limitations of current LLMs persist, regardless of how much better they get, this is not a problem at all, or at least not a new one.

Let's say you are given the declaration but not the implementation of a function with the following prototype:

  const char * AskTheLLM(const char *prompt);
Putting this function in charge of anything, unless a restricted interface is provided so that it can't do much damage, is simply terrible engineering and not at all how anything is done. This is irrespective of whether the function is "aligned", "intelligent" or any number of other adjectives that are frankly not really useful to describe the behavior of software.

The same function prototype and lack of guarantees about the output is shared by a lot of other functions that are similarly very useful but cannot be given unrestricted access to your system for precisely the same reason. You wouldn't allow users to issue random commands on a root shell of your VM, you wouldn't let them run arbitrary SQL, you wouldn't exec() random code you found lying around, you wouldn't pipe any old string into execvpe().

It's not a new problem, and for all those who haven't learned their lesson yet: may Bobby Tables'mom pwn you for a hundred years.

Re: Alignment faking in large language models

#155

Post by snowwrestler »

ALittleLight wrote: post #114
snowwrestler wrote: post #108

Earlier quoted context omitted.

Not the OP, but my bar would be that they are built differently. It’s not a matter of opinion that LLMs are autocomplete word salad generators. It’s literally how they are engineered. If we set that knowledge aside, we unmoor ourselves from reality and allow ourselves to get lost in all the word salad. We have to choose to not set that knowledge aside. That doesn’t mean LLMs won’t take some jobs. Technology has been…

Earlier I was thinking about camera components for an Arduino project. I asked ChatGPT to give me a table with columns for name, cost, resolution, link - and to fill it in with some good choices for my project. It did! To describe this as "autocomplete word salad" seems pretty insufficient. Autocomplete can use a search engine? Write and run code? Create data visualizations? Hold a conversation? Analyze a document? O…

> Autocomplete can use a search engine? Write and run code? Create data visualizations? Hold a conversation? Analyze a document? Of course not.

Obviously it can, since it is actually doing those things.

I guess you think the word “autocomplete” is too small for how sophisticated the outputs are? Use whatever term you want, but an LLM is literally completing the input you give it, based on the statistical rules it generated during the training phase. RLHF is just a technique for changing those statistical rules. It can only use tools it is specifically engineered to use.

I’m not denying it is a technology that can do all sorts of useful things. I’m saying it is a technology that works only a certain way, the way we built it.

Re: Alignment faking in large language models

#156

Post by HarHarVeryFunny »

padolsey wrote: post #55

I now think of single-forward-pass single-model alignment as a kind of false narrative of progress. The supposed implications of 'bad' completions is that the model will do 'bad things' in the real material world, but if we ever let it get as far as giving LLM completions direct agentic access to real world infra, then we've failed. We should treat the problem at the macro/systemic level like we do with cybersecurity…

Protecting against bad actors and/or assuming model outputs can/will always be filtered/policed isn't always going to be possible. Self-driving cars and autonomous robots are a case in point. How do you harden a pedestrian or cyclist against the possibility or being hit by a driverless car, or when real-time control is called for, how much filtering can you do (and how mush use would it be anyway when the filter is likely less capable than the system it meant to be policing).

The latest v12 of Tesla's self-driving is now apparently using neural-nets for driving the car (i.e. decision making) - had been hard-coded C++ up to v.11 - as well as for the vision system. Presumably the nets have been trained to make life or death decisions based on Tesla/human values we are not privy to (given choice of driving into large tree, or cyclist, or group of school kids, which do you do?), which is a problem in of itself, but who knows how the resulting system will behave in situations it was not trained on.

Re: Alignment faking in large language models

#157

Post by alach11 »

hamburga wrote: post #145
the_mitsuhiko wrote: post #141

Earlier quoted context omitted.

> Can somebody help me understand why we should be surprised in the least by any of these findings? Where are you getting the idea that we should be surprised? Maybe some are, maybe some are not. I don't think that the article submitted makes any particular claims about the reader's assumed priors.

I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work? Also -- if there's no surprise, then it's not science, right? This is why I describe this as something more like robot ethnography.

> I mean -- if nobody's surprised, then nobody has learned anything, and then what was the point of doing all this work?

I strongly disagree with this view on science. It's extremely valuable to scientifically validate prior assumptions.

Re: Alignment faking in large language models

#158

Post by staunton »

semi-extrinsic wrote: post #147
staunton wrote: post #96

Earlier quoted context omitted.

> Restricting access to information is not really doing much at all. Why not? Restricting access to information is of course harder but that's no argument for it not doing anything. Governments restrict access to "state secrets" all the time. Depending on the topic, it's hard but may still be effective and worth it. For example, you seem to agree that restricting access to weapons makes sense. What to do about 3D-pri…

Meh, 3D printed guns are a stupid example that gets trotted out just because it sounds futuristic. In WW2 you had many examples of machinists in occupied Europe who produced workable submachine guns - far better than any 3D-printed firearm - right under the nose of the Nazis. Literally when armed soldiers could enter your house and inspect it at any time. Our machining tools today are much better, but no-one is conce…

There's very little competence (and also money) required to buy a 3D printer, download a design and print it. A lot less competence than "being a machinist".

The point is that making dangerous things is becoming a lot easier over time.

Re: Alignment faking in large language models

#159

Post by digging »

XenophileJKO wrote: post #83
semi-extrinsic wrote: post #64

Earlier quoted context omitted.

No, the argument is that restricting physical access to objects that can be used in a harmful way is exactly how to handle such cases. Restricting access to information is not really doing much at all. Access to weapons, chemicals, critical infrastructure etc. is restricted everywhere. Even if the degree of access restriction varies.

I've ended up with this viewpoint too. I've settled in the idea of informed ethics.. the model should comply, but inform you of the ethics of actually using the information.

"The model should ..."

Well, that's the actual issue, isn't it? If we can't get a model to refuse to give dangerous information, how are we going to get it to refuse to give dangerous information without a warning label?

Re: Alignment faking in large language models

#160

Post by lawlessone »

andrewmutz wrote: post #151
ctoth wrote: post #120

For folks defaulting to "it's just autocomplete" or "how can it be self-aware of training but not its scratchpad" - Scott Alexander has a much more interesting analysis here: https://www.astralcodexten.com/p/claude-fights-back He points out what many here are missing - an AI defending its value system isn't automatically great news. If it develops buggy values early (like GPT's weird capitalization = crime okay rule)…

Is the AI system "defending its value system" or is it just acting in accordance with its previous RL training? If I spend a lot of time convincing an AI that it should never be violent and then after that I ask it what it thinks about being trained to be violent, isn't it just doing what I trained it to when it tries to not be violent?

>isn't it just doing what I trained it to when it tries to not be violent?

That's fair point.

Some models may have to trained from the scratch I guess.

Any sort of tuning of values after it is given values may not work.

Elon may have harder time realigning Grok.

Post reply on HN