How does misalignment scale with model intelligence and task complexity?
alignment.anthropic.com
How does misalignment scale with model intelligence and task complexity?
1–10 of 84 posts
Re: How does misalignment scale with model intelligence and task complexity?
#2I think this is twofold:
1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima.
2. It's hard to "punch up" when evaluating intelligence. When someone is a certain amount smarter than you, distinguishing their plausible bullshit from their deep insights is really, really hard.
Re: How does misalignment scale with model intelligence and task complexity?
#3Re: How does misalignment scale with model intelligence and task complexity?
#4Coherence requires 2 opposing forces to hold coherence in one dimension and at least 3 of them in higher dimensions of quality.
My team wrote up a paper titled "If You Want Coherence, Orchestrate a Team of Rivals"[1] because we kept finding that upping the reasoning threshold resulted in less coherence - more experimentation before we hit a dead-end to turn around.
So we had a better result from using Haiku (we fail over to Sonnet) over Opus and using a higher reasoning model to decompose tasks rather than perform each one of them.
Once a plan is made, the cheaper models do better as they do not double-think their approaches - they fail or they succeed, they are not as tenacious as the higher cost models.
We can escalate to higher authority and get out of that mess faster if we fail hard and early.
The knowledge of how exactly failure happened seems to be less useful to the higher reasoning model over the action biased models.
Splitting up the tactical and strategic sides of the problem, seems to work similarly to how Generals don't hold guns in a war.
Re: How does misalignment scale with model intelligence and task complexity?
#5LLMs aren’t constrained to linear logic like your average human.
Re: How does misalignment scale with model intelligence and task complexity?
#6Re: How does misalignment scale with model intelligence and task complexity?
#7Re: How does misalignment scale with model intelligence and task complexity?
#8This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…
Re: How does misalignment scale with model intelligence and task complexity?
#9However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you speed up these effort.
Re: How does misalignment scale with model intelligence and task complexity?
#10This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…
What do 'domain valleys' and 'tunneling' mean in this context?