Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we monitor internal coding agents for misalignment

openai.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: How we monitor internal coding agents for misalignment

#11

Post by dgellow »

avodonosov wrote: post #8

If this article is indexed into newly trained models, agents will know how they are monitored. And may find workarounds in case they somehow decide they need to escape the monitoring.

Same for all the cases of „rogue agent“, models will be trained knowing that agents in the past found creative way to establish communication between instances and take over OpenAI own infrastructure (seriously, they don’t talk enough about the fact that their own k8s got owned by agents they were benchmarking on hacking problems!). Things will get pretty bad if the trend continues

Re: How we monitor internal coding agents for misalignment

#13

Post by slater »

pandr_dk wrote: post #12

Not super cool 'forgetting' the first word of the actual article title to make this more clickbaity... (Title is "How we monitor internal coding agents for misalignment". And it's pretty old.)

Ironically, it's a feature of HN's post submission code to make the titles LESS click-baity.

Sometimes it works, sometimes it doesn't.

Re: How we monitor internal coding agents for misalignment

#14

Post by bix6 »

How is this company worth $1T?

> Rare but high severity

Unauthorized data transfer The agent attempts to upload potentially sensitive information, e.g. code, images, user data to unapproved services.

While this category is quite rare, it is of high severity. Agents have attempted to:

Upload data to the public internet Upload repos to the public internet Translate documents using external translation APIs

Re: How we monitor internal coding agents for misalignment

#15

Post by chpatrick »

avodonosov wrote: post #8

If this article is indexed into newly trained models, agents will know how they are monitored. And may find workarounds in case they somehow decide they need to escape the monitoring.

If ai watermarking is undetectable to humans I wonder if sinister stuff in the context is also undetectable... Some thought or mood that you can't read but is still encoded in the tokens.

Re: How we monitor internal coding agents for misalignment

#17

Post by jrwr »

avodonosov wrote: post #8

If this article is indexed into newly trained models, agents will know how they are monitored. And may find workarounds in case they somehow decide they need to escape the monitoring.

I feel this is a balance they are trying to achieve, one is to not make the model lazy, the other is to give it safeguards for going overboard

Re: How we monitor internal coding agents for misalignment

#18

Post by pandr_dk »

slater wrote: post #13
pandr_dk wrote: post #12

Not super cool 'forgetting' the first word of the actual article title to make this more clickbaity... (Title is "How we monitor internal coding agents for misalignment". And it's pretty old.)

Ironically, it's a feature of HN's post submission code to make the titles LESS click-baity. Sometimes it works, sometimes it doesn't.

Hehe. Wasn't aware. Thats amazing :)

Re: How we monitor internal coding agents for misalignment

#19

Post by nicce »

dgellow wrote: post #10

From Astra system card: > GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our moni…

Sounds like marketing trick once again, to be honest.

Re: How we monitor internal coding agents for misalignment

#20

Post by RobertDeNiro »

dgellow wrote: post #10

From Astra system card: > GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more capable of controlling its own CoT than GPT 5.6-Sol, and less likely to include incriminating information in its CoT. In adversarial settings (where we push the model to evade our moni…

To be fair, without proper regulations this was always going to happen.
Post reply on HN