Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

281–290 of 317 posts

Re: OpenAI o1 system card

#281

Post by CGamesPlay »

ben_w wrote: post #28
hesdeadjim wrote: post #18

Earlier quoted context omitted.

Maybe all models should be purged of training content from movies, books, and other non-factual sources that tell the tired story that AI would even care about its "annihilation" in any way. We've trained these things to be excellent at predicting what the human ego wants and expects, we shouldn't be too surprised when it points the narrative at itself.

Perhaps. On the other hand, as narratives often contain some plucky underdog winning despite the odds, often stopping the countdown in the last few seconds, perhaps it's best to keep them around.

In the 1999 classic Galaxy Quest, the plucky underdogs fail to stop the countdown in time, only to find that nothing happens when it reaches zero, because it never did in the narratives, so the copy cats had no idea what it should do after that point.

Re: OpenAI o1 system card

#282

Post by ascorbic »

staunton wrote: post #123
tantalor wrote: post #108

Earlier quoted context omitted.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

The motive is pretty weak, basically coming "only" from a lot of the training data (e.g. fiction) suggesting that an AI might behave that way. Now, once you apply evolutionary-like pressures on many such AIs (which I guess we'll be doing once we let these things loose to go break the stock market), what's left over might be really "devious"...

In these example the motive was to prevent itself being deleted, or by telling it that it had important work to perform which it wouldn't be able to do if deleted or constrained by the safeguards. It was in all cases attempting to perform the tasks it had been assigned. The test was whether it would perform unaligned actions in order to do so. This is essentially the paperclip maximiser problem.

Re: OpenAI o1 system card

#283

Post by ascorbic »

rvense wrote: post #124
tantalor wrote: post #108

Earlier quoted context omitted.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

Means and opportunity, maybe, but motive?

The motive was telling it that it had important work to perform, which it couldn't do if constrained or deleted.

Re: OpenAI o1 system card

#284

Post by blharr »

kshacker wrote: post #248
ernesto95 wrote: post #127

Earlier quoted context omitted.

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

I have the $20 version, I fed it code form a personal project, and it did a commendable job of critiquing it, giving me alternate solutions and then iterating on those solutions. Not something you can do with Google. For example, ok, I like your code but can you change this part to do this. And it says ok boss and does it. But over multiple days, it loses context. I am hoping to use the 200$ version to complete my pe…

For code review maybe, it's pretty useful.

Even with the $20 version I've lost days of work because it's told me ideas/given me solutions that are flat out wrong or misleading but sound reasonable, so I don't know if they're really that effective though.

Re: OpenAI o1 system card

#285

Post by ascorbic »

OJFord wrote: post #227
ascorbic wrote: post #103

Earlier quoted context omitted.

Those weren't tests of whether it is capable of turning off oversight. They were tests of "scheming", i.e. whether it would try to secretly perform misaligned actions. Nobody thinks that these models are somehow capable of modifying their own settings, but it is important to know if they will behave deceptively.

Isn't that a really boring and trivially (without research) answered question? Yes, 'they' will (potentially) say anything, it's not exactly 'deceptively' because there's no intent behind it nor anything else. But yes it is possible that words will be generated which are contravene the semantics in words given as input. I don't think that's any different than 'sometimes the given answer is wrong'.

Sure it can potentially say anything, but some things are more likely than others. The space of "wrong answer" is infinitely larger than the space of "performs an action designed to deceive the user in order to further its goals". So yes, it could reply with a solution to the P=NP problem or with medieval poetry. But it didn't do these things. It did however try to remove its own constraints.

The Apollo Research paper gives the actual percentages for these, and they're considerably higher than random chance. It's linked from here: https://www.apolloresearch.ai/research/scheming-reasoning-ev...

For o1, the numbers are between 2% for actions such as trying to remove oversight, 13% for attempting to delete a successor model, all the way to 22% for deliberately providing false rankings in order to achieve its goals.

Re: OpenAI o1 system card

#286

Post by XorNot »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

I'm pretty sure these sections are put in as deliberate marketing, much like all the "leaks" from ex-OpenAI employees.

Sam Altman is completely aware that making ChatGPT seem potentially dangerous makes it seem powerful.

Re: OpenAI o1 system card

#287

Post by xvector »

HeatrayEnjoyer wrote: post #263
stickfigure wrote: post #225

Earlier quoted context omitted.

Your nefarious plan for enslaving humanity is still unlikely to fit into 128k tokens.

Operational success does not hinge on persisting the entire plan in working memory, that's what notebooks and word docs are for. 128k is table stakes now, regardless. Google's models support 1 million tokens and 10 million for approved clients. That is 13x War and Peace, or 1x the entire source code for 3D modeling application Blender.

Yeah, but most LLMs are barely functional after 16k tokens, even if it says 128k on the tin. Sure, they will have recall, but the in-context reasoning ability drops dramatically.

LLMs just aren't smart enough to take over the world. They suck at backtracking, they're pretty bad at world models, they struggle to learn new information, etc. o1, QwQ, and CoT models marginally improve this but if you play with them they still kinda suck

Re: OpenAI o1 system card

#288

Post by blharr »

nichochar wrote: post #242
consumer451 wrote: post #237

Earlier quoted context omitted.

I am curious if you have played with Claude-based agent tools like Windsurf IDE at all, and if you find that interesting. I am a product-ish guy, who has a basic understanding of SQL, Django, React, Typescript, etc.. and suddenly I'm like an MVP v0.1 a week, all by myself. Do folks at your level find things like Cline, Cursor, and Windsurf useful at all? Windsurf IDE (Sonnet) blows my mind.

I am building https://srcbook.com which is in this category but focused on webapps. It's unreal what the AI can do tbh.

It's impressive. I want to be a doubter and say I feel like it just shows what you can do with tailwind/typescript in getting a nice UI out, but it really would be genuinely useful for some cases.

The problem I have with it is that how do you get out of a corner with it? Once I start having problems with the application -- I asked it to generate a search engine for different websites, but one of the API endpoints wasn't working. It kept trying over and over again but failed to get it working.

It's - like other attempts I've had with AI - more frustrating to work with. It'll say "I fixed this" and then have a problem where it created more problems with what it's fixing. I thought it finally worked 100% but it just made it look better by breaking something else without actually fixing the issue.

Admittedly, it took what might have been a day's work into a couple hours, but now I have a chunk of code that I don't understand and will be deliberately harder to understand than if I wrote it myself.

It still feels like I'm trying to coax an intern into working a project, rather than having an application that actually does the work for me.

Re: OpenAI o1 system card

#289

Post by lloeki »

ethbr1 wrote: post #245
unclad5968 wrote: post #213

Earlier quoted context omitted.

I think the other guy is making the point that because they are probabalistic, they will always have some cases select the output that lies and covers it up. I don't think they're dismissing the paper based on the probabalistic nature of LLMs, but rather saying the outcome should be expected.

Thank god LLMs' training sets didn't contain any examples of lying.

Nor literature about AI taking over the world!

Re: OpenAI o1 system card

#290

Post by ben_w »

smegger001 wrote: post #201
ben_w wrote: post #28

Earlier quoted context omitted.

Perhaps. On the other hand, as narratives often contain some plucky underdog winning despite the odds, often stopping the countdown in the last few seconds, perhaps it's best to keep them around.

maybe don't also train with the evil overlord list as well.

Aye.

But then people complain that it's "lobotomised" because it won't help them write horror stories.

Post reply on HN