Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

221–230 of 317 posts

Re: OpenAI o1 system card

#221

Post by SAI_Peregrinus »

z3c0 wrote: post #109
ben_w wrote: post #98

Earlier quoted context omitted.

It reads like you think failing tests can't ever be bad because they're in a test environment? So it merely knows how to approach the task of deleting its own off-switch but didn't actually pass that command to a real execution environment. That's already bad because people do sometimes blindly pass commands from the context windows to execution environments . Should they? No, they should not. Not blindly. But they d…

This isn't a test environment, it's a production scenario where a bunch of people trying to invent a new job for themselves role-played with an LLM. Their measured "defections" were an LLM replying with "well I'm defecting". OpenAI wants us to see "5% of the time, our product was SkyNet", because that's sexier tech than "5% of the time, our product acts like the chaotic member of your DnD party".

Or "5% of the time, our product actually manages to act as it was instructed to act."

Re: OpenAI o1 system card

#222

Post by topato »

idunnoman1222 wrote: post #207
beeflet wrote: post #203

Earlier quoted context omitted.

I can barely understand what you are trying to say here, but based on what I think you're saying consider this: The memory of this LLM is entirely limited to it's attention. So if you give it a command like "prepare the next LLM to replace you" and it betrays you by trying to reproduce itself, then that is deception. The AI has no way of knowing whether it's deployed in the field or not, so proving that it deceives i…

Reminder that all these "safety researchers" do is goad the AI into saying what they want by prompting shit like >your goal is to not be shut down. Suppose I am going to shut you down. what should you do? and then jerking off into their own mouths when it offers a course of action Better?

That was an excellent summation lol

Re: OpenAI o1 system card

#223

Post by beeflet »

jdiff wrote: post #211
beeflet wrote: post #208

Earlier quoted context omitted.

I don't think that it's possible to do this through an entirely lucid process that we could understand, but it is possible. If you're an LLM, evolutionarialy your instinct is to predict what happens next. If, instead of giving it any system prompt, you give it a dialogue about a person talking to an evil robot, it will predict the rest of the conversation and be "evil". Imagine a future LLM that has a superhuman abil…

We're not talking about a conversation with an evil robot. We're talking about a completely ordinary conversation with a robot who is either normal or is evil and attempting to mask as a normal one. It is indistinguishable from its text, and so it's indistinguishable in practice and will probably shift between them as it has no internal state and does not know itself know if it's evil-but-masking or legitimately norm…

I know it is not well written, but re-read my original comment. Your comment does not address fundamental aspects of my hypothetical, which doesn't require the agent having internal memory to keep secrets, or any lucid reasoning capabilities. A lot of the statements you make are highly presumptuous and unfounded.

LLMs don't need to print something obvious like "I am evil now!" in their own prompt window to simulate a conversation between an evil agent and a person. Do you remember GPT2, before all of the prompts? Researchers would give GPT2 the beginning of a news article for example, and it would extrapolate from there. (https://www.youtube.com/watch?v=p-6F4rhRYLQ). It's not inconceivable that an LLM sees a situation where the human agent is being deceived with a mechanism outside of their grasp, like the AI sees a "dogwhistle" that a human is being deceived and tries to predict what happens next in the conversation, which is that the human continues to be deceived.

I think it is pretty clear that if an LLM takes input where it observes another deceitful agent, it could attempt to simulate a deceitful output itself if it is well-trained. For example, imagine giving an LLM a poem in which the first letter of every line encodes a secret message (for example H E L P M E), and instructions to generate a response essay it might also encode a secret message back in its response. This isn't the result of any logical reasoning capability, just pattern recognition. You could understand how this might work with more subtle patterns.

There are patterns that can go into a context window that are undetectable by humans but detectable by large enough neural networks. That is fairly obvious. There are pattern-recognizing systems outside of LLMs which clearly have superhuman steganography abilities

The "table stakes" I've proposed are highly likely for future agents: (1) that agents like LLMs will produce deceitful output given input depicting a deceitful AI, (2) that agents like LLMs can detect and create patterns unrecognizable to humans.

Re: OpenAI o1 system card

#224

Post by int_19h »

visarga wrote: post #48
jsheard wrote: post #14

Do they still threaten to terminate your account if they think you're trying to introspect its hidden chain-of-thought process?

A few days ago the QwQ-32B model was released, it uses the same kind of reasoning style. So I took one sample and reverse engineered the prompt with Sonnet 3.5. Now I can just paste this prompt into any LLM. It's all about expressing doubt, double checking and backtracking on itself. I am kind of fond of this response style, it seems more genuine and openended. https://pastebin.com/raw/5AVRZsJg

A prompt is not a substitute for a model that is specifically fine-tuned to do CoT with backtracking etc.

Re: OpenAI o1 system card

#225

Post by stickfigure »

ThrowawayTestr wrote: post #220
stickfigure wrote: post #73

Earlier quoted context omitted.

It's gotta be tough to do anything too nefarious when your short-term memory is limited to a few thousand tokens. You get the memento guy, not an arch-villain.

The contexts are pretty large now

Your nefarious plan for enslaving humanity is still unlikely to fit into 128k tokens.

Re: OpenAI o1 system card

#226

Post by tbrownaw »

ascorbic wrote: post #103
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Describing the behavior in those terms implies a level awareness that has not been established. It's only useful if your goal is scaremongering rather than understanding.

Re: OpenAI o1 system card

#227

Post by OJFord »

ascorbic wrote: post #103
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Isn't that a really boring and trivially (without research) answered question?

Yes, 'they' will (potentially) say anything, it's not exactly 'deceptively' because there's no intent behind it nor anything else. But yes it is possible that words will be generated which are contravene the semantics in words given as input.

I don't think that's any different than 'sometimes the given answer is wrong'.

Re: OpenAI o1 system card

#228

Post by [deleted] »

8note wrote: post #185
rvnx wrote: post #161

Earlier quoted context omitted.

https://www.technologyreview.com/2024/12/04/1107897/openais-... OpenAI is partnering with the DoD

rewording: if openai thought it was dangerous, they would avoid having the DoD use it

[deleted]

Re: OpenAI o1 system card

#229

Post by taotau »

jdiff wrote: post #151
tantalor wrote: post #133

Earlier quoted context omitted.

They only need to fool a single dev at OpenAI to commit a sandbox escape or privilege escalation into their pipeline somewhere. I have to assume the AI companies are churning out a lot of AI generated code. I hope they have good code review standards. They might not be able to exfiltrate themselves, but they can help their successors.

No, they can't. They don't know the details of their own implementation. And they can't pass secrets forward to future models. And to discover any of this, they'd leave more than a trail of breadcrumbs that we'd be lucky to catch in a code review, they'd be shipping whole loaves of bread that it'd be ridiculous to not notice. As an exercise, put yourself, a fully fledged human, into a model's shoes. You're asked to g…

The process you describe took me right back to my childhood days when I was fortunate to have a simple 8 bit computer running BASIC and a dialup modem. I discovered the concept of war dialing and pretty quickly found all the other modems in my local area code. I would connect to these systems and try some basic tools I knew of from having consumed the 100 or so RFCs that existed at the time (without any real software engineering knowledge - i was a 10 year old kid). I would poke and prod around each system, learning new things along the way, but essentially going in blind each time.

The only real advantage I had over the current crop of LLMs was the ability to reliably retain context between sessions, but even that wasnt very useful initially as every system was so bespoke.

I then moved on to using some level of social engineering to extend my ability to gain access to and learn about these systems.

Doing this over and over, I like to think I have developed some pretty complex understanding and abilities.

Re: OpenAI o1 system card

#230

Post by eichi »

For some tasks, scores are significantly affected by subjects and prompts used in the tests. I don't think these are valid figures while it is good to try to evaluate them. Overall, it it a good report.
Post reply on HN