How does misalignment scale with model intelligence and task complexity? • Hacker News Board

Re: How does misalignment scale with model intelligence and task complexity?

#11

Post by crabmusket » Tue, Feb 03, 2026, 1:23 AM UTC

smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably productive?

Re: How does misalignment scale with model intelligence and task complexity?

#12

Post by jmtulloss » Tue, Feb 03, 2026, 1:26 AM UTC

The comments so far seem focused on taking a cheap shot, but as somebody working on using AI to help people with hard, long-term tasks, it's a valuable piece of writing.

- It's short and to the point

- It's actionable in the short term (make sure the tasks per session aren't too difficult) and useful for researchers in the long term

- It's informative on how these models work, informed by some of the best in the business

- It gives us a specific vector to look at, clearly defined ("coherence", or, more fun, "hot mess")

Re: How does misalignment scale with model intelligence and task complexity?

#13

Post by nayroclade » Tue, Feb 03, 2026, 1:28 AM UTC

The models they tested are already way behind the current state-of-the-art. Would be interesting to see if their results hold up when repeated with the latest frontier models.

Re: How does misalignment scale with model intelligence and task complexity?

#14

Post by energy123 » Tue, Feb 03, 2026, 1:29 AM UTC

CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

Incoherence is not error.

You can have a vanishingly small error and an incoherence at its max.

That would be evidence of perfect alignment (zero bias) and very low variance.

Re: How does misalignment scale with model intelligence and task complexity?

#15

Post by charcircuit » Tue, Feb 03, 2026, 1:37 AM UTC

crabmusket wrote: post #11

smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably…

If you are on the same wave length as someone you don't need to produce a full spec. You can trust that the other person has the same vision as you and will pick reasonable ways to implement things. This is one reason why personalized AI agents are important.

Re: How does misalignment scale with model intelligence and task complexity?

#16

Post by [deleted] » Tue, Feb 03, 2026, 1:41 AM UTC

[deleted]

Re: How does misalignment scale with model intelligence and task complexity?

#17

Post by [deleted] » Tue, Feb 03, 2026, 1:45 AM UTC

esyir wrote: post #10

xanderlewis wrote: post #8

Earlier quoted context omitted.

What do 'domain valleys' and 'tunneling' mean in this context?

Not the OP, but my interpretation here is that if you model the replies as some point in a vector space, assuming points from a given domain cluster close to each other, replies that span two domains need to "tunnel" between these two spaces.

[deleted]

Re: How does misalignment scale with model intelligence and task complexity?

#18

Post by p-e-w » Tue, Feb 03, 2026, 2:05 AM UTC

CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

> When someone is a certain amount smarter than you, distinguishing their plausible bullshit from their deep insights is really, really hard.

Insights are “deep” not on their own merit, but because they reveal something profound about reality. Such a revelation is either testable or not. If it’s testable, distinguishing it from bullshit is relatively easy, and if it’s not testable even in principle, a good heuristic is to put it in the bullshit category by default.

Re: How does misalignment scale with model intelligence and task complexity?

#19

Post by skydhash » Tue, Feb 03, 2026, 2:05 AM UTC

crabmusket wrote: post #11

smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably…

Programming languages can be a thinking tool for a lot of tasks. Very much like a lot of notation, like music sheet and map drawing. A condensed and somewhat formal manner of describing ideas can increase communication speed. It may lack nuance, but in some case, nuance is harmful.

The nice thing about code compared to other notation is that it's useful on its. You describe an algorithm and the machine can then solve the problem ad infinitum. It's one step instead of the two step of writing a spec and having an LLM translate it, then having to verify the output and alter it.

Assembly and high level languages are equivalent in terms of semantics. The latter helps in managing complexity, by reducing harmful possibilities (managing memory, off-by-one errors) and presenting common patterns (iterators/collections, struct and other data structures, ....) so that categories of problems are easily solved. There's no higher level of computing model unlocked. Just faster level of productivity unlocked by following proven patterns.

Spec driven workflow is a mirage, because even the best specs will leave a lot of unspecified details. Which are crucial as most of programming is making the computer not do the various things it can do.

Re: How does misalignment scale with model intelligence and task complexity?

#20

Post by skydhash » Tue, Feb 03, 2026, 2:13 AM UTC

p-e-w wrote: post #18

CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

> When someone is a certain amount smarter than you, distinguishing their plausible bullshit from their deep insights is really, really hard. Insights are “deep” not on their own merit, but because they reveal something profound about reality. Such a revelation is either testable or not. If it’s testable, distinguishing it from bullshit is relatively easy, and if it’s not testable even in principle, a good heuristic…

The issue is the revelation. It's always individual at some level. And don't forget our senses are crude. The best way is to store "insights" as information until we collect enough data that we can test it again (hopefully without a lot of bias). But that can be more than a lifetime work, so sometimes you have to take some insights at face value based on heuristics (parents, teachers, elder, authority,...)