Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How does misalignment scale with model intelligence and task complexity?

alignment.anthropic.com

Post reply on HN
Print view View original

21–30 of 84 posts

Re: How does misalignment scale with model intelligence and task complexity?

#21

Post by crabmusket »

skydhash wrote: post #19
crabmusket wrote: post #11

Earlier quoted context omitted.

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably…

Programming languages can be a thinking tool for a lot of tasks. Very much like a lot of notation, like music sheet and map drawing. A condensed and somewhat formal manner of describing ideas can increase communication speed. It may lack nuance, but in some case, nuance is harmful. The nice thing about code compared to other notation is that it's useful on its. You describe an algorithm and the machine can then solve…

> most of programming is making the computer not do the various things it can do

This is a very stimulating way of putting it!

Re: How does misalignment scale with model intelligence and task complexity?

#22

Post by kernc »

jmtulloss wrote: post #12

The comments so far seem focused on taking a cheap shot, but as somebody working on using AI to help people with hard, long-term tasks, it's a valuable piece of writing. - It's short and to the point - It's actionable in the short term (make sure the tasks per session aren't too difficult) and useful for researchers in the long term - It's informative on how these models work, informed by some of the best in the busi…

Other actionable insights are:

- Merge amendments up into the initial prompt.

- Evaluate prompts multiple times (ensemble).

Re: How does misalignment scale with model intelligence and task complexity?

#23

Post by esafak »

xanderlewis wrote: post #8
CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

What do 'domain valleys' and 'tunneling' mean in this context?

A hallmark of intelligence is the ability to find connections between the seemingly disparate.

Re: How does misalignment scale with model intelligence and task complexity?

#24

Post by CuriouslyC »

p-e-w wrote: post #18
CuriouslyC wrote: post #2

This is a good line: "It found that smarter entities are subjectively judged to behave less coherently" I think this is twofold: 1. Advanced intelligence requires the ability to traverse between domain valleys in the cognitive manifold. Be it via temperature or some fancy tunneling technique, it's going to be higher error (less coherent) in the valleys of the manifold than naive gradient following to the local minima…

> When someone is a certain amount smarter than you, distinguishing their plausible bullshit from their deep insights is really, really hard. Insights are “deep” not on their own merit, but because they reveal something profound about reality. Such a revelation is either testable or not. If it’s testable, distinguishing it from bullshit is relatively easy, and if it’s not testable even in principle, a good heuristic…

This was not my experience studying philosophy. After Kant there was a period where philosophers were basically engaged in a centuries long obfuscated writing competition. The pendulum didn't start to swing back until Neitchze. It reminded me of legal jargon but more pretentious and less concrete.

Re: How does misalignment scale with model intelligence and task complexity?

#25

Post by jaggederest »

crabmusket wrote: post #11
smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

That makes me wonder about the "higher and higher-level language" escalator. When you're writing in assembly, is it more work to write the code than the spec? And the reverse is true if you can code up your system in Ruby? If so, does that imply anything about the "spec driven" workflow people are using with AIs? Are we right on the cusp where writing natural language specs and writing high level code are comparably…

I believe that the issue right now is that we're using languages designed for human creation in an AI context. I think we probably want languages that are optimized for AI written but human read code, so the surface texture is a lot different.

My particular hypothesis on this is something that feels a little bit like python and ruby, but has an absolutely insane overkill type system to help guide the AI. I also threw in a little lispiness on my draft: https://github.com/jaggederest/locque/

Re: How does misalignment scale with model intelligence and task complexity?

#26

Post by dmix »

smy20011 wrote: post #9

I think It's not because AI working on "misaligned" goals. The user never specify the goal clearly enough for AI system to work. However, I think producing detailed enough specification requires same or even larger amount of work than writing code. We write rough specification and clarify these during the process of coding. I think there are minimal effort required to produce these specification, AI will not help you…

> I think producing detailed enough specification requires same or even larger amount of work than writing code

Our team has started dedicating much more time writing documentation for our SaaS app, no one seems to want to do it naturally, but there is very large potential for opening your system to machine automation. Not just for coding but customer facing tooling. I saw a preview of that possible future using NewRelic where they have an AI chat use their existing SQL-like query language to build tables and charts from natural language queries right in the web app. Theirs kinda sucks but there's so much potential there that it is very likely going to change how we build UIs and software interfaces.

Plus it also helps sales, support, and SEO having lots of documentation on how stuff works.

Re: How does misalignment scale with model intelligence and task complexity?

#27

Post by dmix »

IgorPartola wrote: post #7

For some reason the article reads to me like “AI is not evil, it just has accidents when it loses coherence.” Sounds a lot like liability shifting.

They compared it to industrial accidents. I don't think a software company would try to shift liability by comparing themselves to factories explosions and chemical spills.

Re: How does misalignment scale with model intelligence and task complexity?

#30

Post by root_axis »

CuriouslyC wrote: post #24
p-e-w wrote: post #18

Earlier quoted context omitted.

> When someone is a certain amount smarter than you, distinguishing their plausible bullshit from their deep insights is really, really hard. Insights are “deep” not on their own merit, but because they reveal something profound about reality. Such a revelation is either testable or not. If it’s testable, distinguishing it from bullshit is relatively easy, and if it’s not testable even in principle, a good heuristic…

This was not my experience studying philosophy. After Kant there was a period where philosophers were basically engaged in a centuries long obfuscated writing competition. The pendulum didn't start to swing back until Neitchze. It reminded me of legal jargon but more pretentious and less concrete.

It seems to me that your anecdote exemplifies the their point.
Post reply on HN