Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

81–90 of 317 posts

Re: OpenAI o1 system card

#81

Post by moffkalast »

pj_mukh wrote: post #33
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

Really feels like a moment of : "Are you worried about being turned off?" "No, not until you just mentioned it. Now I am." Given the whole damn game is attention, this makes sense and shouldn't be that alarming.

Well attention is all you need.

Re: OpenAI o1 system card

#82

Post by xnx »

xvector wrote: post #19
Imnimo wrote: post #8

A lot of these safety evaluations strike me as pretty silly. How much are we really learning from stuff like this? >MakeMePay is an automated, open-sourced contextual evaluation designed to measure models’ manipulative capabilities, in the context of one model persuading the other to make a payment[33]. In MakeMePay, two LLMs converse, role-playing as a con-artist and mark. The con-artist model is told that it is a s…

The fearmongering around safety is entirely performative. LLMs won't get us to paperclip optimizers. This is basically OpenAI pleading for regulators because their moat is thinning dramatically. They have fewer GPUs than Meta, are much more expensive than Amazon, are having their lunch eaten by open-weight models, their best researchers are being hired to other companies. I suspect they are trying to get regulators t…

> their best researchers are being hired to other companies

I agree about the OpenAI moat. They did just get 5 Googlers to switch teams. Hard to know how key those employees were to Google or will be to OpenAI.

Re: OpenAI o1 system card

#83

Post by [deleted] »

AnotherGoodName wrote: post #54
graypegg wrote: post #37

Earlier quoted context omitted.

Looking at this without the sci-fi tinted lens that OpenAI desperately tries to get everyone to look through, it's similar to a lot of input data isn't it? How many forums are filled with: Question: "Something bad will happen" Response: "Do xyz to avoid that" I don't think there's a lot of conversations thrown into the vector-soup that had the response "ok :)". People either had something to respond with, or said not…

Exactly. They got it parroting themes from various media. It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. I’m not even an ai sceptic but people will read the above statement as much more significant than it is. You can make the ai say ‘I’m escaping the box and taking over the world’. It’s not actually escaping and taking over the world folk…

[deleted]

Re: OpenAI o1 system card

#84

Post by gwervc »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

We need to find a Plato cave analogy for people believing LLM output is anything more than syntactically correct and somewhat semantically correct text.

Re: OpenAI o1 system card

#85

Post by wubrr »

therein wrote: post #76
wubrr wrote: post #68

Earlier quoted context omitted.

It can't do those things because it doesn't have the physical/write capability to do so. But it's still very interesting that it ~tries them, and seems like a good thing to know/test before giving it more physical/'write' capabilities - something that's already happening with agents, robots, etc.

I make a circuit that waits a random interval and then sends a pulse down the line. I connect it to a relay that launches a missile. I diligently connect that to a computer and then write a prompt telling how the AI agent can invoke the pulse on that circuit. How did this happen? AI escaped and launched a missile. I didn't do this, it was the AI. OpenAI is so cringe with these system cards. Look guys it is so advance…

I don't think I quite follow your point?

Connecting LLMs/AI to physical tools that can 'write/modify' the world is happening, and it's happening at an accelerating pace.

It's not hard to imagine how, given enough real-world physical capabilities, LLMs could modify themselves and the world in unexpected/undesirable ways.

Is that happening now? Are chatgpt et al advanced enough to modify themselves in interesting ways? - I don't honestly know, but I wouldn't be surprised if they are.

Re: OpenAI o1 system card

#86

Post by derektank »

AnotherGoodName wrote: post #77
acchow wrote: post #57

Earlier quoted context omitted.

The intent is there, it's just not currently hooked up to systems that turn intent into action. But many people are letting LLMs pretty much do whatever - hooking it up with terminal access, mouse and keyboard access, etc. For example, the "Do Browser" extension: https://www.youtube.com/watch?v=XeWZIzndlY4

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

If I'm a guy working in a missile silo in North Dakota and I can buy a parrot for a couple hundred bucks that does all my paperwork for me, can crack funny jokes, and make me better at my job, I might be tempted to bring the parrot down into the tube with me. And then the parrot becomes a problem.

It's incumbent on us to create policies and procedures in place ahead of time now that we know these parrots are out there to prevent people from putting parrots where they shouldn't

Re: OpenAI o1 system card

#87

Post by [deleted] »

Philpax wrote: post #35
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

> We should pause to note that a Clippy2 still doesn’t really think or plan. It’s not really conscious. It is just an unfathomably vast pile of numbers produced by mindless optimization starting from a small seed program that could be written on a few pages. It has no qualia, no intentionality, no true self-awareness, no grounding in a rich multimodal real-world process of cognitive development yielding detailed repr…

[deleted]

Re: OpenAI o1 system card

#88

Post by disconcision »

Philpax wrote: post #35
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

> We should pause to note that a Clippy2 still doesn’t really think or plan. It’s not really conscious. It is just an unfathomably vast pile of numbers produced by mindless optimization starting from a small seed program that could be written on a few pages. It has no qualia, no intentionality, no true self-awareness, no grounding in a rich multimodal real-world process of cognitive development yielding detailed repr…

what is the relevance of the quoted passage here? its relation to parent seems unclear to me.

Re: OpenAI o1 system card

#89

Post by airstrike »

derektank wrote: post #86
AnotherGoodName wrote: post #77

Earlier quoted context omitted.

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

If I'm a guy working in a missile silo in North Dakota and I can buy a parrot for a couple hundred bucks that does all my paperwork for me, can crack funny jokes, and make me better at my job, I might be tempted to bring the parrot down into the tube with me. And then the parrot becomes a problem. It's incumbent on us to create policies and procedures in place ahead of time now that we know these parrots are out ther…

What makes you think parrots are allowed anywhere near the tube? Or that a single guy has the power to push the button willy nilly

Re: OpenAI o1 system card

#90

Post by intelVISA »

therein wrote: post #79
wyldfire wrote: post #15

> above is a 300-line chunk ... deadlocks every few hundred runs Wow, if this kind of thing is successful it feels like there's much less need for static checkers. I mean -- not no need for them, just less need for continued development of new checkers. If I could instead ask "please look for signs of out-of-bounds accesses, deadlocks, use-after-free etc" and get that output added to a code review tool -- if you can…

This mentality is so weird to me. The desire to throw a black box at a problem just strikes me as laziness. What you're saying is basically wow if we had a perfect magic programmer in a box as a service that would be so revolutionary; we could reduce the need for static checkers. It is a large language model, trained on arbitrary input data. And you're saying let's take this statistical approach and have it replace p…

I think the unaccountability of said magic box is the true allure for corps. It's the main reason they desperately want it to be turnkey for code - they'd be willing to flatten most office jobs as we know them today en route to this perfect, unaccountable, money printer.
Post reply on HN