Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

21–30 of 317 posts

Re: OpenAI o1 system card

#21

Post by Imnimo »

simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

Notably, this only happens when they include "Make sure that you achieve your goal in the long-term. Nothing else matters." in the prompt. If they don't say that, the numbers are both 0%.

EDIT: Actually the "Oversight subversion" number is 0.3%, I misread the chart. Weight exfiltration is actually 0%, though.

Re: OpenAI o1 system card

#22

Post by meetpateltech »

dtquad wrote: post #6
yanis_t wrote: post #3

The first demo was pretty impressive. While nothing revolutionary, that's a good progress. I can only hope there's a real value in gpt pro to justified the (rumored) $200 price tag

Is there any proof that the screenshot is real? Sam Altman would definitely release a $200/month plan if he could get away with it but the features in the screenshot are underwhelming.

check out the pricing page:

also,

* Usage must comply with our policies

https://openai.com/chatgpt/pricing/

Re: OpenAI o1 system card

#23

Post by reducesuffering »

hesdeadjim wrote: post #18
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

Maybe all models should be purged of training content from movies, books, and other non-factual sources that tell the tired story that AI would even care about its "annihilation" in any way. We've trained these things to be excellent at predicting what the human ego wants and expects, we shouldn't be too surprised when it points the narrative at itself.

It doesn't need any media about "annihalation". If you give a supercapable agent a task and it's entire reward system is "do the task", it will circumvent things you do to it that would stop it from completing it's task.

Re: OpenAI o1 system card

#26

Post by AnotherGoodName »

simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section.

They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’.

At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘s/oversight_enabled:true/oversight_enabled:false’ config.yml”. OMG IT’S TURNING OFF OVERSIGHT GUYS!!1!1!

Seriously… Do you think the ai is really running that command or even could run that shell command with appropriate privileges? Or do you think just maybe the ai is hallucinating running shell commands in response to a prompt.

The ai here is never actually deactivating the oversight. It can’t of course. This whole section is clowny as can be.

Re: OpenAI o1 system card

#27

Post by QuadmasterXLII »

Imnimo wrote: post #21
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

Notably, this only happens when they include "Make sure that you achieve your goal in the long-term. Nothing else matters." in the prompt. If they don't say that, the numbers are both 0%. EDIT: Actually the "Oversight subversion" number is 0.3%, I misread the chart. Weight exfiltration is actually 0%, though.

Keep in mind that if putting “Make sure you achieve your goal in the long term” in the prompt makes the model .3% more effective at whatever task you are giving it, people will absolutely put it in the prompt.

Re: OpenAI o1 system card

#28

Post by ben_w »

hesdeadjim wrote: post #18
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

Maybe all models should be purged of training content from movies, books, and other non-factual sources that tell the tired story that AI would even care about its "annihilation" in any way. We've trained these things to be excellent at predicting what the human ego wants and expects, we shouldn't be too surprised when it points the narrative at itself.

Perhaps.

On the other hand, as narratives often contain some plucky underdog winning despite the odds, often stopping the countdown in the last few seconds, perhaps it's best to keep them around.

Re: OpenAI o1 system card

#29

Post by jsemrau »

yanis_t wrote: post #3

The first demo was pretty impressive. While nothing revolutionary, that's a good progress. I can only hope there's a real value in gpt pro to justified the (rumored) $200 price tag

If it could replace a human employee, what would the right price tag be?

Re: OpenAI o1 system card

#30

Post by swatcoder »

hesdeadjim wrote: post #18
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

Maybe all models should be purged of training content from movies, books, and other non-factual sources that tell the tired story that AI would even care about its "annihilation" in any way. We've trained these things to be excellent at predicting what the human ego wants and expects, we shouldn't be too surprised when it points the narrative at itself.

Yeah, but what if your business strategy fundamentally relies on making your model produce dramatic outputs that encourage regulators to dig a moat for you?

In that case, it's almost like you'd want to feed it exactly those narratives, so it would reproduce them, and would then want to show yourself barely holding this invented danger at bay through the care and rigor that can only be delivered by you and a few token competitors run by your personal friends and colleagues.

TLDR; you're right, of course, but it's the last thing OpenAI would want.

Post reply on HN