Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

71–80 of 317 posts

Re: OpenAI o1 system card

#71

Post by jsheard »

AnotherGoodName wrote: post #54
graypegg wrote: post #37

Earlier quoted context omitted.

Looking at this without the sci-fi tinted lens that OpenAI desperately tries to get everyone to look through, it's similar to a lot of input data isn't it? How many forums are filled with: Question: "Something bad will happen" Response: "Do xyz to avoid that" I don't think there's a lot of conversations thrown into the vector-soup that had the response "ok :)". People either had something to respond with, or said not…

Exactly. They got it parroting themes from various media. It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is. I’m not even an ai sceptic but people will read the above statement as much more significant than it is. You can make the ai say ‘I’m escaping the box and taking over the world’. It’s not actually escaping and taking over the world folk…

> It’s really hard to read this as anything other than a desperate attempt to pretend the ai is more capable than it really is.

Tale as old as time, they've been doing this since GPT-2 which they said was "too dangerous to release".

Re: OpenAI o1 system card

#72

Post by aftbit »

shagie wrote: post #45
griomnib wrote: post #36

Earlier quoted context omitted.

It almost definitely ingested hundreds of books, short stories, and film and television scripts from various online sites in the “machine goes rogue genre” which is fairly large. It’s pretty much just an autocomplete of War Games, The Matrix, Neuromancer, and every other cyber-dystopian fiction.

The Freeze-Frame Revolution by Peter Watts was one of the books recommended to me on this subject. And even saying much more than that may be a spoiler. I also recommend the book.

I'll second that recommendation. It's relatively short and enjoyable, at least when compared to a lot of Peter Watts.[1] I'd really like to read the obvious sequel that the ending sets up.

1: Don't get me wrong, I loved the Behemoth series and Blindsight, but they made me feel very dark. This one... is still a bit dark, but less so IMO.

Re: OpenAI o1 system card

#73

Post by stickfigure »

gbear605 wrote: post #67
hypeatei wrote: post #64

Earlier quoted context omitted.

What are people legitimately worried about LLMs doing by themselves? I hate to reduce them to "just putting words together" but that's all they're doing. We should be more worried about humans treating LLM output as truth and using it to, for example, charge someone with a crime.

People are already just hooking LLMs up to terminals with web access and letting them go. Right now they’re too dumb to do something serious with that, but text access to a terminal is certainly sufficient to do a lot of bad things in the world.

It's gotta be tough to do anything too nefarious when your short-term memory is limited to a few thousand tokens. You get the memento guy, not an arch-villain.

Re: OpenAI o1 system card

#74

Post by griomnib »

shagie wrote: post #45
griomnib wrote: post #36

Earlier quoted context omitted.

It almost definitely ingested hundreds of books, short stories, and film and television scripts from various online sites in the “machine goes rogue genre” which is fairly large. It’s pretty much just an autocomplete of War Games, The Matrix, Neuromancer, and every other cyber-dystopian fiction.

The Freeze-Frame Revolution by Peter Watts was one of the books recommended to me on this subject. And even saying much more than that may be a spoiler. I also recommend the book.

Thanks for the recc.

Re: OpenAI o1 system card

#75

Post by konschubert »

gbear605 wrote: post #67
hypeatei wrote: post #64

Earlier quoted context omitted.

What are people legitimately worried about LLMs doing by themselves? I hate to reduce them to "just putting words together" but that's all they're doing. We should be more worried about humans treating LLM output as truth and using it to, for example, charge someone with a crime.

People are already just hooking LLMs up to terminals with web access and letting them go. Right now they’re too dumb to do something serious with that, but text access to a terminal is certainly sufficient to do a lot of bad things in the world.

Millions of people are hooked up to a terminal as well.

Re: OpenAI o1 system card

#76

Post by therein »

wubrr wrote: post #68
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

It can't do those things because it doesn't have the physical/write capability to do so. But it's still very interesting that it ~tries them, and seems like a good thing to know/test before giving it more physical/'write' capabilities - something that's already happening with agents, robots, etc.

I make a circuit that waits a random interval and then sends a pulse down the line. I connect it to a relay that launches a missile. I diligently connect that to a computer and then write a prompt telling how the AI agent can invoke the pulse on that circuit.

How did this happen? AI escaped and launched a missile. I didn't do this, it was the AI.

OpenAI is so cringe with these system cards. Look guys it is so advanced.

Re: OpenAI o1 system card

#77

Post by AnotherGoodName »

acchow wrote: post #57
AnotherGoodName wrote: post #26

Earlier quoted context omitted.

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

The intent is there, it's just not currently hooked up to systems that turn intent into action. But many people are letting LLMs pretty much do whatever - hooking it up with terminal access, mouse and keyboard access, etc. For example, the "Do Browser" extension: https://www.youtube.com/watch?v=XeWZIzndlY4

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

Re: OpenAI o1 system card

#78

Post by refulgentis »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

I don't think this is a helpful tone.

Do they think it's _actually_ a real script deactivating oversight? Well, no, you know that. I can only assume it's a rhetorical device.

Simultaneously, these things are given access to actual things and will be more and more. It's important to measure and observe these things. It's completely novel to have an AI, with a hidden trace the user can't see, and it turns out that hidden trace will include things like it actively deceiving and doing things it was told not to do.

Re: OpenAI o1 system card

#79

Post by therein »

wyldfire wrote: post #15

> above is a 300-line chunk ... deadlocks every few hundred runs Wow, if this kind of thing is successful it feels like there's much less need for static checkers. I mean -- not no need for them, just less need for continued development of new checkers. If I could instead ask "please look for signs of out-of-bounds accesses, deadlocks, use-after-free etc" and get that output added to a code review tool -- if you can…

This mentality is so weird to me. The desire to throw a black box at a problem just strikes me as laziness.

What you're saying is basically wow if we had a perfect magic programmer in a box as a service that would be so revolutionary; we could reduce the need for static checkers.

It is a large language model, trained on arbitrary input data. And you're saying let's take this statistical approach and have it replace purpose made algorithms.

Let's get rid of motion tracking and rotoscoping capabilities in Adobe After Effects. Generative AI seems to handle it fine. Who needs to create 3D models when you can just describe what you want and then AI just imagines it?

Hey AI, look at this code, now generate it without my memory leaks and deadlocks and use-after-free? People who thought about these problems mindfully and devised systematic approaches to solving them must be spinning in their graves.

Re: OpenAI o1 system card

#80

Post by SirMaster »

JTyQZSnP3cQGa8B wrote: post #66
SirMaster wrote: post #51

Earlier quoted context omitted.

It can't today, but if it's smart enough how do you know it wouldn't be able to in the future?

> The question of whether machines can think is about as relevant as the question of whether submarines can swim It's a program with a lot of data running on a big calculator. It won't ever be "smart."

Sure, but is it so implausible that it could some day have the knowledge to perhaps exploit some security hole to run some code that does do things like disable things or exfiltrate data etc?
Post reply on HN