Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does misalignment scale with model intelligence and task complexity?

alignment.anthropic.com

Post reply on HN
Print view View original

51–60 of 84 posts

Re: How does misalignment scale with model intelligence and task complexity?

#51

Post by gf000 »

jaggederest wrote: post #25
crabmusket wrote: post #11

Earlier quoted context omitted.

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably…

I believe that the issue right now is that we're using languages designed for human creation in an AI context. I think we probably want languages that are optimized for AI written but human read code, so the surface texture is a lot different. My particular hypothesis on this is something that feels a little bit like python and ruby, but has an absolutely insane overkill type system to help guide the AI. I also threw…

I don't know, LLMs strive on human text, so I would wager that a language designed for humans would quite closely match an ideal one for LLMs. Probably the only difference is that LLMs are not "lazy", they better tolerate boilerplate, and lower complexity structures likely fit them better. (E.g. they can't really one-shot understand some imported custom operator that is not very common in its training data)

Also, they rely surprisingly closely on "good" code patterns, like comments and naming conventions.

So if anything, a managed language [1] with a decent type system and not a lot of features would be the best, especially if it has a lot of code in its training data. So I would rather vote on Java, or something close.

[1] reasoning about life times, even if aided by the compiler is a global property, and LLMs are not particularly good at that

Re: How does misalignment scale with model intelligence and task complexity?

#53

Post by gylterud »

TonyStr wrote: post #47
esafak wrote: post #23

Earlier quoted context omitted.

A hallmark of intelligence is the ability to find connections between the seemingly disparate.

Does this make conspiracy theorists highly intelligent?

No, but they emulate intelligence by making up connections between seemingly disparate things, where there are none.

Re: How does misalignment scale with model intelligence and task complexity?

#54

Post by w10-1 »

esafak wrote: post #23
xanderlewis wrote: post #8

Earlier quoted context omitted.

What do 'domain valleys' and 'tunneling' mean in this context?

A hallmark of intelligence is the ability to find connections between the seemingly disparate.

Actually, a hallmark could be to prune illusory connections, right? That would decrease complexity rather than amplifying it.

Re: How does misalignment scale with model intelligence and task complexity?

#55

Post by Nevermark »

gopalv wrote: post #46
Nevermark wrote: post #40

Earlier quoted context omitted.

> Coherence requires 2 opposing forces This seems very basic to any kind of information processing beyond straight shot predictable transforms. Expansion and reduction of possibilities, branches, scope, etc. Biological and artificial neural networks converging into multiple signals, that are reduced by competition between them. Scientific theorizing, followed by experimental testing. Evolutionary genetic recombinatio…

> This seems very basic Yes, this is not some sort of hard-fought wisdom. It should be common sense, but I still see a lot of experiments which measure the sound of one hand clapping. In some sense, it is a product of laziness to automate human supervision with more agents, but on the other hand I can't argue with the results. If you don't really want the experiments and data from the academic paper, we have a white…

> Yes, this is not some sort of hard-fought wisdom.

By basic I didn't mean uninteresting.

In fact, despite the pervasiveness and obviousness of the control and efficiency benefits of push-pull, generating-reducing, cooperation-competition, etc., I don't think I have ever seen any kind of general treatment or characterization that pulled all these similar dynamics together. Or a hierarchy of such.

> In some sense, it is a product of laziness to automate human supervision with more agents, but on the other hand I can't argue with the results.

I think it is the fact that the agents are operating coherently with the respective complementary goals. Whereas, asking one agent to both solve and judge creates conflicting constraints before a solution has begun.

Creative friction.

I am reminded of brainstorming sessions, where it is so important to note ideas, but not start judging them, since who knows what crazy ideas will fit or spark together. Later they can be selected down.

So we institutionalize this separation/staging with human teams too, even if it is just one of us (within our context limits, over two inference sessions :).

Re: How does misalignment scale with model intelligence and task complexity?

#56

Post by bob1029 »

You simply can't have a single shot context with so many simultaneous constraints and expect to make forward progress. This cannot be solved with additional silicon, power or data.

Smaller prompts and fewer tools tends to be more stable. I try to stay within 1000 tokens and 10 tools for a single inference pass. I become visibly amused when I read many of the system prompts out there. Anthropomorphism is the biggest anti pattern with these models. It's a very easy and comfortable trap to fall into.

The core issue I see with coding agents is that the moment you read a file, you've polluted the context in terms of token coherence. It's probably not critical in most cases, but it's safer to pretend like it is. Recursive/iterative decomposition of the problem is the only thing I've seen so far that can scale arbitrarily. For example, if you invoke a sub agent every time you read a file, you can reduce the impact to the token budget of the caller by orders of magnitude. The callee can return a brief summary or yes/no response to the caller after reading 500kb of source. This applies at each level of recursion and can compound dramatically (exponentially) over just a few nested calls.

Re: How does misalignment scale with model intelligence and task complexity?

#58

Post by reverius42 »

booleandilemma wrote: post #49
CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

> the ability to traverse between domain valleys in the cognitive manifold. Couldn't you have just said "know about a lot of different fields"? Was your comment sarcastic or do you actually talk like that?

I think they mean both "know about a lot of different fields" and also "be able to connect them together to draw inferences", the latter perhaps being tricky?

Re: How does misalignment scale with model intelligence and task complexity?

#59

Post by sandos »

kernc wrote: post #22
jmtulloss wrote: post #12

The comments so far seem focused on taking a cheap shot, but as somebody working on using AI to help people with hard, long-term tasks, it's a valuable piece of writing. - It's short and to the point - It's actionable in the short term (make sure the tasks per session aren't too difficult) and useful for researchers in the long term - It's informative on how these models work, informed by some of the best in the busi…

Other actionable insights are: - Merge amendments up into the initial prompt. - Evaluate prompts multiple times (ensemble).

Sometimes when I was stressed, I have used several models to verify each others´ work. They usually find problems, too!

This is very useful for things that take time to verify, we have CI stuff that takes 2-3 hours to run and I hate when those fails because of a syntax error.

Re: How does misalignment scale with model intelligence and task complexity?

#60

Post by ithkuil »

esafak wrote: post #23
xanderlewis wrote: post #8

Earlier quoted context omitted.

What do 'domain valleys' and 'tunneling' mean in this context?

A hallmark of intelligence is the ability to find connections between the seemingly disparate.

The word "seemingly" is doing a lot of work here.

Sometimes things that look very different actually are represented with similar vectors in latent space.

When that happens to us it "feels like" intuition; something you can't really put a finger on and might require work to put into a form that can be transferred to another human that has a different mental model

Post reply on HN