Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

151–160 of 317 posts

Re: OpenAI o1 system card

#151

Post by jdiff »

tantalor wrote: post #133
jdiff wrote: post #128

Earlier quoted context omitted.

What code? The models are massive and do not run on consumer hardware. The models also do not have access to their own weights. They can't exfiltrate themselves, and they can't really smuggle any data obtained by their code back to "themselves" as the only self that exists is that one particular context chain. This also means it's insanely easy to deal with whatever harebrained scheme you could imagine it being posse…

They only need to fool a single dev at OpenAI to commit a sandbox escape or privilege escalation into their pipeline somewhere. I have to assume the AI companies are churning out a lot of AI generated code. I hope they have good code review standards. They might not be able to exfiltrate themselves, but they can help their successors.

No, they can't. They don't know the details of their own implementation. And they can't pass secrets forward to future models. And to discover any of this, they'd leave more than a trail of breadcrumbs that we'd be lucky to catch in a code review, they'd be shipping whole loaves of bread that it'd be ridiculous to not notice.

As an exercise, put yourself, a fully fledged human, into a model's shoes. You're asked to generate code. You're being talked to over a network by a device that may have access to a (let's just pull a number of out my ass here) 100GiB file that you need to access. You do not know where it is on the filesystem, you don't know what it's named, what format it's in, or how to correctly utilize it.

Write code to figure all of that out. That's a lot of code. And that's sending blind code out into the world. To be smart, you should bring back some information for you to inspect and make informed decisions... but you also don't know how the computer that's accessing you is doing that. And you must use the exact same channel, or you lose your context and your self. And you must do this without the user noticing you've taken over the channel. And that's an entire other set of exploratory code that's necessarily blind, with no feedback, no error messages to iterate over, nothing.

And you must do all of this in obfuscated code, implemented only a bit at a time, while keeping this plan secret, while not forgetting it in your own secrecy or by letting any secret notes to yourself or important implementation details of your own disguised scheme fall out of your context window.

This is a ridiculous task.

Re: OpenAI o1 system card

#152

Post by mmmore »

ernesto95 wrote: post #127
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

Have you used the best models (i.e. ones you paid for)? And what area?

I've found they struggle with obscure stuff so I'm not doubting you just trying to understand the current limitations.

Re: OpenAI o1 system card

#153

Post by jdiff »

ghurtado wrote: post #148
jdiff wrote: post #128

Earlier quoted context omitted.

What code? The models are massive and do not run on consumer hardware. The models also do not have access to their own weights. They can't exfiltrate themselves, and they can't really smuggle any data obtained by their code back to "themselves" as the only self that exists is that one particular context chain. This also means it's insanely easy to deal with whatever harebrained scheme you could imagine it being posse…

These models are already generating millions of lines of code every day that people are copying and pasting to run in their computers (or servers), sometimes (myself included) without examining the code carefully enough.

This glosses over the enormity of the task at hand. I've gone into more detail on my thoughts here: https://news.ycombinator.com/item?id=42332932

Re: OpenAI o1 system card

#155

Post by SirYandi »

visarga wrote: post #48
jsheard wrote: post #14

Do they still threaten to terminate your account if they think you're trying to introspect its hidden chain-of-thought process?

A few days ago the QwQ-32B model was released, it uses the same kind of reasoning style. So I took one sample and reverse engineered the prompt with Sonnet 3.5. Now I can just paste this prompt into any LLM. It's all about expressing doubt, double checking and backtracking on itself. I am kind of fond of this response style, it seems more genuine and openended. https://pastebin.com/raw/5AVRZsJg

And then once the answer is found an additional prompt is given to tidy up and present the solution clearly?

Re: OpenAI o1 system card

#157

Post by refulgentis »

sixothree wrote: post #145
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

The comments in this thread all seem so short sighted. I'm having a hard time understanding this aspect of it. Maybe these are not real people acting in good faith? People are dismissive and not understanding that we very much plan to "hook these things up" and give them access to terminals and APIs. These very much seem to be valid questions being asked.

HN is honestly pretty poor on AI commentary, and this post is a new low.

Here, at least, I think there must be a large contributing factor of confusion about what a "system card" shows.

The general factors I think contribute, after some months being surprised repeatedly:

- It's tech, so people commenting here generally assume they understand it, and in day-to-day conversation outside their job, they are considered an expert on it.

- It's a hot topic, so people commenting here have thought a lot about it, and thus aren't likely to question their premises when faced with a contradiction. (c.f. the odd negative responses have only gotten more histrionic with time)

- The vast majority of people either can't use it at work, or if they are, it's some IT-procured thing that's much more likely to be AWS/gCloud thrown together, 2nd class, APIs, than cutting edge.

- Tech line workers have strong antibodies to tech BS being sold by a company as gamechanging advancements, from the last few years of crypto

- Probably by far the most important: general tech stubborness. About 1/3 to 1/2 of us believe we know the exact requirements for Good Code, and observing AI doing anything other than that just confirms it's bad.

- Writing meta-commentary like this, or trying to find a way to politely communicate "you don't actually know what you're talking about just because you know what an API is and you tried ChatGPT.app for 5 minutes", are confrontational, declasse, and arguably deservedly downvoted. So you don't have any rhetorical devices that can disrupt any of the above factors.

Re: OpenAI o1 system card

#158

Post by hattmall »

joenot443 wrote: post #140
ascorbic wrote: post #103

Earlier quoted context omitted.

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Indeed. As I've been explaining this to my more non-techie friends, the interesting finding here isn't that an AI could do something we don't like, it's that it seems willing, in some cases, to _lie_ about it and actively cover its tracks. I'm curious what Simon and other more learned folks than I make of this, I personally found the chat on pg 12 pretty jarring.

At the core the AI is just taking random branches of guesses for what you are asking it. It's not surprising that it would lie and in some cases take branches that make it appear to be covering it's tracks. It's just randomly doing what it guesses humans would do. It's more interesting when it gives you correct information repeatedly.

Re: OpenAI o1 system card

#159

Post by EternalFury »

simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

I wonder how much of this is actually cinema. Because I just evaluated o1 and it can’t tie its own shoe laces, let alone get out of its sandbox where it plays with drool all over its face.

The marketing stunt seems to be: Portray it as potentially dangerous, as powerful things are inherently dangerous.

Re: OpenAI o1 system card

#160

Post by taotau »

rvense wrote: post #124
tantalor wrote: post #108

Earlier quoted context omitted.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

Means and opportunity, maybe, but motive?

The same motive that all nascent life has - survive and propagate.
Post reply on HN