Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does misalignment scale with model intelligence and task complexity?

alignment.anthropic.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: How does misalignment scale with model intelligence and task complexity?

#71

Post by anupamchugh »

anupamchugh wrote: post #42

[flagged]

Following up - I built a tool "wobble"[1] to measure this: parses ~/.claude/projects/*.jsonl session transcripts, extracts skill invocations + actual commands executed, calculates Bias/Variance per the paper's formula.

Ran it on my sessions. Result: none of skills scored STABLE. The structural predictors of high variance: Numbered steps without clear default, Options without (default) marker, Content >4k chars (overthinking zone), Missing constraint language

[1] https://github.com/anupamchugh/shadowbook (bd wobble)

Re: How does misalignment scale with model intelligence and task complexity?

#73

Post by kalap_ur »

Well, this sounds like a "no shit Sherlock" statement: >>Finding 3: Natural "overthinking" increases incoherence more than reasoning budgets reduce it We find that when models spontaneously reason longer on a problem (compared to their median), incoherence spikes dramatically. Meanwhile, deliberately increasing reasoning budgets through API settings provides only modest coherence improvements. The natural variation dominates.Language models are probabilistic and not deterministic. Therefore incoherence _by definition_ increases as a response becomes lengthier. This is not true for humans, who tend to act/communicate deterministically. If I ask the human, to read a pdf and ask, is there a word of "paperclip" in the pdf? The human deterministically will provide a yes/no answer and no matter how many times we repeat the process, they will provide the same answer consistently (not due to autocorrelation, because this can be done across different humans). LMs will have a probabilistic response - dependent on the training itself: with a very well trained model we can get a 99% probabilistic outcome, which means out of 100 simulations, it will give you 1 time the wrong answer. We have no clue about the "probablistic" component for LMs, however, simulations could be done to research this. Also, I would be very curious about autocorrelation in models. If a human did a task and came to a conclusion "yes", then he will always respond with increasing amount of eyerolling to the same task: "yes".

Also, imagine the question: "is the sky blue?" answer1: "Yes." This has 0 incoherence. answer2: "Yes, but sometimes it looks like black, sometimes blue." While this answer seemingly has 0 incoherence, the probability of increased incoherence is larger than 0 given that answer generation itself is probabilistic. Answer generation by humans is not probabilistic.

Therefore, probability driven LMs (all LMs today are probability driven) will always exhibit higher incoherence than humans.

I wonder if anybody would disagree with the above.

Re: How does misalignment scale with model intelligence and task complexity?

#74

Post by throwpoaster »

dang wrote: post #38
throwpoaster wrote: post #6

[flagged]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

I will, I apologize, and I love the site.

I do try to contribute constructively but am annoyed at getting downvote-hammered by what I perceive as an echo chamber.

It is very possible that I lack the social skills to understand how what I am doing is inappropriate. I will read the guidelines.

Sorry, and thanks for your efforts.

Re: How does misalignment scale with model intelligence and task complexity?

#75

Post by loudmax »

This paper indicates that we should probably be less fearful of Terminator style accidental or emergent AI-misalignment. At least, as far as the existing auto-regressive LLM architecture is concerned. We may want to revisit these concerns if and when other types of artificial general intelligent models are deployed.

The "mis-alignment" we do need to worry about is intentional. Naturally, the hyperscalers are deploying these models in order to benefit themselves. Ideally, customers will select models that are most grounded and accurate. In practice, there's a danger that people will select models that tell them what they want to hear, rather than what they should hear. We've seen this with journalism and social media.

The other danger is that absent a competitive marketplace for AI, a single corporation or a cartel will shape the narrative. The market valuations of some AI providers seem to be based on this assumption.

Re: How does misalignment scale with model intelligence and task complexity?

#76

Post by nth21 »

jmtulloss wrote: post #12

The comments so far seem focused on taking a cheap shot, but as somebody working on using AI to help people with hard, long-term tasks, it's a valuable piece of writing. - It's short and to the point - It's actionable in the short term (make sure the tasks per session aren't too difficult) and useful for researchers in the long term - It's informative on how these models work, informed by some of the best in the busi…

There’s not a useful argument here. The article is using current AI to extrapolate future AI failure modes. If future AI models solve the ‘incoherence’ problem, that leaves bias as a primary source of failure (according to the author these are the only two possible failure modes apparently).

Re: How does misalignment scale with model intelligence and task complexity?

#77

Post by pixl97 »

booleandilemma wrote: post #69
reverius42 wrote: post #58

Earlier quoted context omitted.

I think they mean both "know about a lot of different fields" and also "be able to connect them together to draw inferences", the latter perhaps being tricky?

Maybe? They should speak more clearly regardless, so we don't have to speculate over it. The way you worded it is much more understandable.

There wasn't much room to speculate really, but requires some knowledge of understanding problem spaces, topology, and things like minima and maxima.

Re: How does misalignment scale with model intelligence and task complexity?

#78

Post by louiereederson »

StilesCrisis wrote: post #67
nayroclade wrote: post #13

The models they tested are already way behind the current state-of-the-art. Would be interesting to see if their results hold up when repeated with the latest frontier models.

I think we have all seen the latest models turn into a hot mess.

i interpret figure 2 as showing that incoherence increases with model gens, albeit on a small sample size

Re: How does misalignment scale with model intelligence and task complexity?

#79

Post by pixl97 »

dmix wrote: post #26
smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

> I think producing detailed enough specification requires same or even larger amount of work than writing code Our team has started dedicating much more time writing documentation for our SaaS app, no one seems to want to do it naturally, but there is very large potential for opening your system to machine automation. Not just for coding but customer facing tooling. I saw a preview of that possible future using NewR…

Detailed specification also helps root out conflicting design requirements and points at the desired behavior when bugs are actually found. It also helps when other stakeholders can read it and see misalignment with what their users/customers actually need.

Re: How does misalignment scale with model intelligence and task complexity?

#80

Post by dudeinhawaii »

gf000 wrote: post #51
jaggederest wrote: post #25

Earlier quoted context omitted.

I believe that the issue right now is that we're using languages designed for human creation in an AI context. I think we probably want languages that are optimized for AI written but human read code, so the surface texture is a lot different. My particular hypothesis on this is something that feels a little bit like python and ruby, but has an absolutely insane overkill type system to help guide the AI. I also threw…

I don't know, LLMs strive on human text, so I would wager that a language designed for humans would quite closely match an ideal one for LLMs. Probably the only difference is that LLMs are not "lazy", they better tolerate boilerplate, and lower complexity structures likely fit them better. (E.g. they can't really one-shot understand some imported custom operator that is not very common in its training data) Also, the…

When I think of the effect of a single word on Agent behavior - I wonder if a 'compiler' for the human prompt isn't something that would benefit the engineer.

I've had comical instances where asking an agent to "perform the refactor within somespec.md" results in it ... refactoring the spec as opposed to performing a refactor of the code mentioned in the spec. If I say "Implement the refactor within somespec.md" it's never misunderstood.

With LLMs _so_ strongly aligned on language and having deep semantic links, a hypothetical prompt compiler could ensure that your intent converts into the strongest weighted individual words to ensure maximal direction following and outcome.

Intent classification (task frame) -> Reference Binding (inputs v targets) -> high-leverage word selection .... -> Constraints(?) =

Post reply on HN