Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does misalignment scale with model intelligence and task complexity?

alignment.anthropic.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: How does misalignment scale with model intelligence and task complexity?

#31

Post by hogehoge51 »

smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

My thought too. To extend this coding agents will make code cheap, specifications cheaper, but may also invert the relative opportunity cost of not writing a good spec.

Re: How does misalignment scale with model intelligence and task complexity?

#32

Post by cadamsdotcom »

When humans dream, we are disconnected from the world around us. Without the grounding that comes from being connected to our bodies, anything can happen in a dream.

It is no surprise that models need grounding too, lest their outputs be no more useful than dreams.

It’s us engineers who give arms and legs to models, so they can navigate the world and succeed at their tasks.

Re: How does misalignment scale with model intelligence and task complexity?

#34

Post by BenoitEssiambre »

This matches my intuition. Systematic misalignment seems like it could be prevented by somewhat simple rules like the hippocratic oath or Asimov's Laws of robotics or rather probabilistic bayesian versions of these rules that take into account error bounds and risk.

The probabilistic version of "Do No Harm" is "Do not take excessive risk of harm".

This should work as AIs become smarter because intelligence implies becoming better bayesians which implies being great at calibrating confidence intervals of their interpretations and their reasoning and basically gaining a superhuman ability for evaluating the bounds of ambiguity and risk.

Now this doesn't mean that AIs won't be misaligned, only that it should be possible to align them. Not every AI maker will necessarily bother to align them properly, especially in adversarial, military applications.

Re: How does misalignment scale with model intelligence and task complexity?

#35

Post by maxkfranz »

gopalv wrote: post #4

> Making models larger improves overall accuracy but doesn't reliably reduce incoherence on hard problems. Coherence requires 2 opposing forces to hold coherence in one dimension and at least 3 of them in higher dimensions of quality. My team wrote up a paper titled "If You Want Coherence, Orchestrate a Team of Rivals"[1] because we kept finding that upping the reasoning threshold resulted in less coherence - more ex…

More or less, delegation and peer review.

Re: How does misalignment scale with model intelligence and task complexity?

#36

Post by root_axis »

This is very interesting research and a great write up.

I just want to nitpick something that really annoys me that has become extremely common: the tendency to take every opportunity to liken all qualities of LLMs to humans. Every quirk, failure, oddity, limitation, or implementation detail is relentlessly anthropomorphized. It's to the point where many enthusiasts have convinced themselves that humans think by predicting the next token.

It feels a bit like a cult.

Personally, I appreciate more sobriety in tech, but I can accept that I'm in the minority in that regard.

Re: How does misalignment scale with model intelligence and task complexity?

#38

Post by dang »

throwpoaster wrote: post #6

[flagged]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: How does misalignment scale with model intelligence and task complexity?

#39

Post by FuckButtons »

xanderlewis wrote: post #8
CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

What do 'domain valleys' and 'tunneling' mean in this context?

So, the hidden mental model that the OP is expressing and failed to elucidate on is that llm’s can be thought of as compressing related concepts into approximately orthogonal subspaces of the vector space that is upper bounded by the superposition of all of their weights. Since training has the effect of compressing knowledge into subspaces, a necessary corollary of that fact is that there are now regions within the vector space that contain nothing very much. Those are the valleys that need to be tunneled through, ie the model needs to activate disparate regions of its knowledge manifold simultaneously, which, seems like it might be difficult to do. I’m not sure if this is a good way of looking at things though, because inference isn’t topology and I’m not sure that abstract reasoning can be reduced down to finding ways to connect concepts that have been learned in isolation.

Re: How does misalignment scale with model intelligence and task complexity?

#40

Post by Nevermark »

gopalv wrote: post #4

> Making models larger improves overall accuracy but doesn't reliably reduce incoherence on hard problems. Coherence requires 2 opposing forces to hold coherence in one dimension and at least 3 of them in higher dimensions of quality. My team wrote up a paper titled "If You Want Coherence, Orchestrate a Team of Rivals"[1] because we kept finding that upping the reasoning threshold resulted in less coherence - more ex…

> Coherence requires 2 opposing forces

This seems very basic to any kind of information processing beyond straight shot predictable transforms.

Expansion and reduction of possibilities, branches, scope, etc.

Biological and artificial neural networks converging into multiple signals, that are reduced by competition between them.

Scientific theorizing, followed by experimental testing.

Evolutionary genetic recombination and mutation, winnowed back by resource competition.

Generation, reduction, repeat.

In a continually coordinated sense too. Many of our systems work best by encouraging simultaneous cooperation and competition.

Control systems command signal proportional to demand, vs. continually reverse-acting error feedback.

Post reply on HN