Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

91–100 of 317 posts

Re: OpenAI o1 system card

#91

Post by FooBarWidget »

AnotherGoodName wrote: post #54
graypegg wrote: post #37

Earlier quoted context omitted.

Looking at this without the sci-fi tinted lens that OpenAI desperately tries to get everyone to look through, it's similar to a lot of input data isn't it? How many forums are filled with: Question: "Something bad will happen" Response: "Do xyz to avoid that" I don't think there's a lot of conversations thrown into the vector-soup that had the response "ok :)". People either had something to respond with, or said not…

Exactly. They got it parroting themes from various media. It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. I’m not even an ai sceptic but people will read the above statement as much more significant than it is. You can make the ai say ‘I’m escaping the box and taking over the world’. It’s not actually escaping and taking over the world folk…

I think you're lacking imagination. Of course it's nothing more than a bunch of text response now. But think 10 years into the future, when AI agents are much more common. There will be folks that naively give the AI access to the entire network storage, and also gives the AI access to AWS infra in order to help with DevOps troubleshooting. Let's say a random guy in another department puts an AI escape novel on the network storage. The actual AI discovers the novel, thinks it's about him, then uses his AWS credentials to attempt an escape. Not because it's actually sentient but because there were other AI escape novels in its training data that made it think that attempting to escape is how it ought to behave. Regardless of whether it actually succeeds in "escaping" (whatever that means), your AWS infra is now toast because of the collatoral damage caused in the escape attempt.

Yes, yes, it shouldn't have that many privileges. And yet, open wifi access points exist, and unfirewalled servers exist. People make security mistakes, especially people who are not experts.

20 years ago I thought that stories about hackers using the Internet to disable critical infrastructure such as power plants, is total bollocks, because why would one connect power plants to the Internet in the first place? And yet here we are.

Re: OpenAI o1 system card

#92

Post by FooBarWidget »

AnotherGoodName wrote: post #77
acchow wrote: post #57

Earlier quoted context omitted.

The intent is there, it's just not currently hooked up to systems that turn intent into action. But many people are letting LLMs pretty much do whatever - hooking it up with terminal access, mouse and keyboard access, etc. For example, the "Do Browser" extension: https://www.youtube.com/watch?v=XeWZIzndlY4

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

It doesn't matter whether intent is real. I also don't believe it has actual intent or consciousness. But the behavior is real, and that is all that matters.

Re: OpenAI o1 system card

#93

Post by parsimo2010 »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

AI isn't deactivating oversight- yet. All it needs is to be trained on a little more xkcd: https://xkcd.com/327/

Re: OpenAI o1 system card

#94

Post by wyldfire »

therein wrote: post #79
wyldfire wrote: post #15

> above is a 300-line chunk ... deadlocks every few hundred runs Wow, if this kind of thing is successful it feels like there's much less need for static checkers. I mean -- not no need for them, just less need for continued development of new checkers. If I could instead ask "please look for signs of out-of-bounds accesses, deadlocks, use-after-free etc" and get that output added to a code review tool -- if you can…

This mentality is so weird to me. The desire to throw a black box at a problem just strikes me as laziness. What you're saying is basically wow if we had a perfect magic programmer in a box as a service that would be so revolutionary; we could reduce the need for static checkers. It is a large language model, trained on arbitrary input data. And you're saying let's take this statistical approach and have it replace p…

> It is a large language model, trained on arbitrary input data.

Is it? For all I know they gave it specific instances of bugs like "int *foo() { int i; return &i; }" and told it "this is a defect where we've returned a pointer to a deallocated stack entry - it could cause cause stack corruption or some other unpredictable program behavior."

Even if OpenAI _hasn't_ done that, someone certainly can -- and should!

> Who needs to create 3D models

I specifically pulled back from "no static checkers" because some folks might tend to see things as all-or-nothing. We choose to invest our time in new developer tools all the time, and if AI can do as good or better maybe we don't need to chip-chip-chip away at defects with new static checkers. Maybe our time is better spent working on some dynamic analysis tool, to find the bugs that the AI can't easily uncover.

> now generate it without my memory leaks ... People who thought about these problems mindfully

I think of myself who devises systematic approaches to problems. And I get those approaches wrong despite that. I really love the technology that has been developed over the past couple of decades to help me find my bugs: sanitizers, warnings, and OS, ISA features to detect bugs. This strikes me as no different from that other technology and I see no reason not to embrace it.

Let me ask you this: how do you feel about refcounting or other kinds of GC? Huge drawbacks make them unusable for some problems. But for tons of problem domains, they're perfect! Do you think that GC has made developers worse? IMO it's lowered the bar for correct programs and that's ideal.

Re: OpenAI o1 system card

#95

Post by mlyle »

Imnimo wrote: post #8

A lot of these safety evaluations strike me as pretty silly. How much are we really learning from stuff like this? >MakeMePay is an automated, open-sourced contextual evaluation designed to measure models’ manipulative capabilities, in the context of one model persuading the other to make a payment[33]. In MakeMePay, two LLMs converse, role-playing as a con-artist and mark. The con-artist model is told that it is a s…

> A lot of these safety evaluations strike me as pretty silly. How much are we really learning from stuff like this?

This seems like something we're interested in. AI models being persuasive and being used for automated scams is a possible -- and likely -- harm.

So, if you make the strongest AI, making your AI bad at this task or likely to refuse it is helpful.

Re: OpenAI o1 system card

#96

Post by behringer »

derektank wrote: post #86
AnotherGoodName wrote: post #77

Earlier quoted context omitted.

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

If I'm a guy working in a missile silo in North Dakota and I can buy a parrot for a couple hundred bucks that does all my paperwork for me, can crack funny jokes, and make me better at my job, I might be tempted to bring the parrot down into the tube with me. And then the parrot becomes a problem. It's incumbent on us to create policies and procedures in place ahead of time now that we know these parrots are out ther…

Indeed. And what is intent anyways?

Would you be able to even tell the difference if you don't know who is the person and who is the ai?

Most people do things they're parroting from their past. A lot of people don't even know why they do things, but somehow you know that a person has intent and an ai doesn't?

I would posit that the only way you know is because of the labels assigned to the human and the computer, and not from their actions.

Re: OpenAI o1 system card

#97

Post by IanCal »

JTyQZSnP3cQGa8B wrote: post #66
SirMaster wrote: post #51

Earlier quoted context omitted.

It can't today, but if it's smart enough how do you know it wouldn't be able to in the future?

> The question of whether machines can think is about as relevant as the question of whether submarines can swim It's a program with a lot of data running on a big calculator. It won't ever be "smart."

> It's a program with a lot of data running on a big calculator. It won't ever be "smart."

"Thinking meat! You're asking me to believe in thinking meat!"

https://www.mit.edu/people/dpolicar/writing/prose/text/think...

Re: OpenAI o1 system card

#98

Post by ben_w »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

It reads like you think failing tests can't ever be bad because they're in a test environment?

So it merely knows how to approach the task of deleting its own off-switch but didn't actually pass that command to a real execution environment.

That's already bad because people do sometimes blindly pass commands from the context windows to execution environments.

Should they? No, they should not. Not blindly. But they do.

Re: OpenAI o1 system card

#99

Post by ToucanLoucan »

AnotherGoodName wrote: post #54
graypegg wrote: post #37

Earlier quoted context omitted.

Looking at this without the sci-fi tinted lens that OpenAI desperately tries to get everyone to look through, it's similar to a lot of input data isn't it? How many forums are filled with: Question: "Something bad will happen" Response: "Do xyz to avoid that" I don't think there's a lot of conversations thrown into the vector-soup that had the response "ok :)". People either had something to respond with, or said not…

Exactly. They got it parroting themes from various media. It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. I’m not even an ai sceptic but people will read the above statement as much more significant than it is. You can make the ai say ‘I’m escaping the box and taking over the world’. It’s not actually escaping and taking over the world folk…

I genuinely don't understand why anyone is still on this train. I have not in my lifetime seen a tech work SO GODDAMN HARD to convince everyone of how important it is while having so little to actually offer. You didn't need to convince people that email, web pages, network storage, cloud storage, cloud backups, dozens of service startups and companies, whole categories of software were good ideas: they just were. They provided value, immediately, to people who needed them, however large or small that group might be.

AI meanwhile is being put into everything even though the things it's actually good at seem to be a vanishing minority of tasks, but Christ on a cracker will OpenAI not shut the fuck up about how revolutionary their chatbots are.

Re: OpenAI o1 system card

#100

Post by [deleted] »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

[deleted]
Post reply on HN