OpenAI o1 system card

Post by snapcaster » Thu, Dec 05, 2024, 8:27 PM UTC

stickfigure wrote: post #73

gbear605 wrote: post #67

Earlier quoted context omitted.

People are already just hooking LLMs up to terminals with web access and letting them go. Right now they’re too dumb to do something serious with that, but text access to a terminal is certainly sufficient to do a lot of bad things in the world.

It's gotta be tough to do anything too nefarious when your short-term memory is limited to a few thousand tokens. You get the memento guy, not an arch-villain.

Post by ascorbic » Thu, Dec 05, 2024, 8:30 PM UTC

AnotherGoodName wrote: post #26

simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

Post by mattangriffel » Thu, Dec 05, 2024, 8:31 PM UTC

disconcision wrote: post #88

Philpax wrote: post #35

Earlier quoted context omitted.

> We should pause to note that a Clippy2 still doesn’t really think or plan. It’s not really conscious. It is just an unfathomably vast pile of numbers produced by mindless optimization starting from a small seed program that could be written on a few pages. It has no qualia, no intentionality, no true self-awareness, no grounding in a rich multimodal real-world process of cognitive development yielding detailed repr…

what is the relevance of the quoted passage here? its relation to parent seems unclear to me.

Post by ben_w » Thu, Dec 05, 2024, 8:34 PM UTC

jsheard wrote: post #71

AnotherGoodName wrote: post #54

Earlier quoted context omitted.

Exactly. They got it parroting themes from various media. It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. I’m not even an ai sceptic but people will read the above statement as much more significant than it is. You can make the ai say ‘I’m escaping the box and taking over the world’. It’s not actually escaping and taking over the world folk…

> It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. Tale as old as time, they've been doing this since GPT-2 which they said was "too dangerous to release".

Post by z3c0 » Thu, Dec 05, 2024, 8:35 PM UTC

ben_w wrote: post #98

AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

It reads like you think failing tests can't ever be bad because they're in a test environment? So it merely knows how to approach the task of deleting its own off-switch but didn't actually pass that command to a real execution environment. That's already bad because people do sometimes blindly pass commands from the context windows to execution environments . Should they? No, they should not. Not blindly. But they d…