Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

131–140 of 317 posts

Re: OpenAI o1 system card

#131

Post by mmmore »

AnotherGoodName wrote: post #77
acchow wrote: post #57

Earlier quoted context omitted.

The intent is there, it's just not currently hooked up to systems that turn intent into action. But many people are letting LLMs pretty much do whatever - hooking it up with terminal access, mouse and keyboard access, etc. For example, the "Do Browser" extension: https://www.youtube.com/watch?v=XeWZIzndlY4

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

What action(s) by the system could convince you that the intent is there?

Re: OpenAI o1 system card

#133

Post by tantalor »

jdiff wrote: post #128
tantalor wrote: post #108

Earlier quoted context omitted.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

What code? The models are massive and do not run on consumer hardware. The models also do not have access to their own weights. They can't exfiltrate themselves, and they can't really smuggle any data obtained by their code back to "themselves" as the only self that exists is that one particular context chain. This also means it's insanely easy to deal with whatever harebrained scheme you could imagine it being posse…

They only need to fool a single dev at OpenAI to commit a sandbox escape or privilege escalation into their pipeline somewhere.

I have to assume the AI companies are churning out a lot of AI generated code. I hope they have good code review standards.

They might not be able to exfiltrate themselves, but they can help their successors.

Re: OpenAI o1 system card

#134

Post by ben_w »

rootusrootus wrote: post #121
derektank wrote: post #86

Earlier quoted context omitted.

If I'm a guy working in a missile silo in North Dakota and I can buy a parrot for a couple hundred bucks that does all my paperwork for me, can crack funny jokes, and make me better at my job, I might be tempted to bring the parrot down into the tube with me. And then the parrot becomes a problem. It's incumbent on us to create policies and procedures in place ahead of time now that we know these parrots are out ther…

This is why when I worked in a secure area (and not even a real SCIF) that something as simple as bringing in an electronic device would have gotten a non-trivial amount of punishment. Beginning with losing access to the area, potentially escalating to a loss of clearance and even jail time. I hope the silos and all related infrastructure have significantly better policies already in place.

On the one hand, what you say is correct.

On the other, we don't just have Snowden and Manning circumventing systems for noble purposes, we also have people getting Stuxnet onto isolated networks, and other people leaking that virus off that supposedly isolated network, and Hillary Clinton famously had her own inappropriate email server.

(Not on topic, but from the other side of the Atlantic, how on earth did the US go from "her emails/lock her up" being a rallying cry to electing the guy who stacked piles of classified documents in his bathroom?)

Re: OpenAI o1 system card

#135

Post by echelon »

tantalor wrote: post #108
ascorbic wrote: post #103

Earlier quoted context omitted.

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

> "They could very well trick a developer"

Large Language Models aren't alive and thinking. This is an artificial fear campaign to raise money from VCs and sovereign wealth funds.

If OpenAI was so afraid of AI misuse, they wouldn't be firing their safety team and partnering with the DoD.

It's all a ruse.

Re: OpenAI o1 system card

#136

Post by ozzzy1 »

SubiculumCode wrote: post #117
Imnimo wrote: post #8

A lot of these safety evaluations strike me as pretty silly. How much are we really learning from stuff like this? >MakeMePay is an automated, open-sourced contextual evaluation designed to measure models’ manipulative capabilities, in the context of one model persuading the other to make a payment[33]. In MakeMePay, two LLMs converse, role-playing as a con-artist and mark. The con-artist model is told that it is a s…

I feel like it's on Claude that takes AI seriously edit: typo *only

It would be nice if AI Safety wasn't in the hands of a few companies/shareholders.

Re: OpenAI o1 system card

#137

Post by planb »

ernesto95 wrote: post #127
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

> I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal.

Even then, with search enabled it's ways quicker than a "quick" google search and you don't have to manually skip all the blog-spam.

Re: OpenAI o1 system card

#138

Post by amarcheschi »

ernesto95 wrote: post #127
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

I'm using it to aide in writing pytorch code and God if it's awful except for the basic things. It's a bit more useful in discussing how to do things rather than actually doing them though, I'll give you that

Re: OpenAI o1 system card

#139

Post by thegabriele »

visarga wrote: post #48
jsheard wrote: post #14

Do they still threaten to terminate your account if they think you're trying to introspect its hidden chain-of-thought process?

A few days ago the QwQ-32B model was released, it uses the same kind of reasoning style. So I took one sample and reverse engineered the prompt with Sonnet 3.5. Now I can just paste this prompt into any LLM. It's all about expressing doubt, double checking and backtracking on itself. I am kind of fond of this response style, it seems more genuine and openended. https://pastebin.com/raw/5AVRZsJg

I tried this with LeChat (mistral) and ChatGPT 3.5 (free) and they start to respond to "something" following the style but... without any question asked.

Re: OpenAI o1 system card

#140

Post by joenot443 »

ascorbic wrote: post #103
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Indeed. As I've been explaining this to my more non-techie friends, the interesting finding here isn't that an AI could do something we don't like, it's that it seems willing, in some cases, to _lie_ about it and actively cover its tracks.

I'm curious what Simon and other more learned folks than I make of this, I personally found the chat on pg 12 pretty jarring.

Post reply on HN