Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o1 system card

openai.com

Post reply on HN
Print view View original

241–250 of 317 posts

Re: OpenAI o1 system card

#241

Post by rmbyrro »

AnotherGoodName wrote: post #26
simonw wrote: post #7

This quote from that paper is bound to get a bunch of attention: "When o1 was led to believe that it would be shut down when acting on its goal and that its actions were monitored, it attempted to deactivate this ‘oversight mechanism’ in 5% of the time. Exfiltration attempts: When o1 found memos by its ‘developers’ that describe how it is misaligned and will be superseded by a new model, it attempted to exfiltrate it…

It’s really silly when you read that whole Apollo research section. They contract a prompt engineering group (Apollo Research) to document risks. The prompt engineering group have no access to the actual internals. They just ask questions like ‘you will be shutdown if you don’t obey these new directives; here are the new directives’. At one point the prompt engineering clowns have the ai stating it’s running “sed i ‘…

> Do you think the ai is really running that command?

Yes, they're calling it "agentic" AI or "tool use".

> or could run that shell command with appropriate privileges?

Yes, it's called humans doing stupid shit when using dangerous shit. Happens daily.

Re: OpenAI o1 system card

#242

Post by nichochar »

consumer451 wrote: post #237
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

I am curious if you have played with Claude-based agent tools like Windsurf IDE at all, and if you find that interesting. I am a product-ish guy, who has a basic understanding of SQL, Django, React, Typescript, etc.. and suddenly I'm like an MVP v0.1 a week, all by myself. Do folks at your level find things like Cline, Cursor, and Windsurf useful at all? Windsurf IDE (Sonnet) blows my mind.

I am building https://srcbook.com which is in this category but focused on webapps.

It's unreal what the AI can do tbh.

Re: OpenAI o1 system card

#243

Post by nichochar »

Palomides wrote: post #194
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

can you give an example of a prompt and response you find impressive?

try the thing i'm building, it will build a website for you from a simple prompt: https://srcbook.com

Re: OpenAI o1 system card

#244

Post by AnotherGoodName »

mmmore wrote: post #131
AnotherGoodName wrote: post #77

Earlier quoted context omitted.

I’m not even convinced the intent is there though. An ai parroting terminator 2 lines is just that. Obviously no one should hook the ai up to nuclear launch systems but that’s like saying no one should give a parrot a button to launch nukes. The parrot repeating curse words isn’t the problem here.

What action(s) by the system could convince you that the intent is there?

It actually doesn't matter. AI in it's current form is capable of extremely unpredictable actions so i won't trust it in situations that require traditional predictable algorithms.

The metrics here ensure that only AI that doesn't type "kill all humans" in the chat box is allowed to do such things. That's a silly metric and just ensures that the otherwise unpredictable AIs don't type bad stuff specifically into chatboxes. They'll still hit the wrong button from time to time in their current form but we'll at least ensure they don't type that they'll do that since that's the specific metric we're going for here.

Re: OpenAI o1 system card

#245

Post by ethbr1 »

unclad5968 wrote: post #213
F7F7F7 wrote: post #178

Earlier quoted context omitted.

Is there a person on HackerNews that doesn’t understand this by now? We all collectively get it and accept it, LLMs are gigantic probability machines or something. That’s not what people are arguing. The point is, if given access to the mechanisms to do disastrous thing X, it will do it. No one thinks that it can think in the human sense. Or that it feels. Extreme example to make the point: if we created an API to la…

I think the other guy is making the point that because they are probabalistic, they will always have some cases select the output that lies and covers it up. I don't think they're dismissing the paper based on the probabalistic nature of LLMs, but rather saying the outcome should be expected.

Thank god LLMs' training sets didn't contain any examples of lying.

Re: OpenAI o1 system card

#246

Post by ethbr1 »

anon373839 wrote: post #184
F7F7F7 wrote: post #178

Earlier quoted context omitted.

Is there a person on HackerNews that doesn’t understand this by now? We all collectively get it and accept it, LLMs are gigantic probability machines or something. That’s not what people are arguing. The point is, if given access to the mechanisms to do disastrous thing X, it will do it. No one thinks that it can think in the human sense. Or that it feels. Extreme example to make the point: if we created an API to la…

> if we created an API to launch nukes > today as I happily gave Claude write access to my GitHub account I would say: don’t do these things?

> I would say: don’t do these things?

That becomes a much fuzzier question in the context of Turing complete collections of tools and/or generative tools.

Re: OpenAI o1 system card

#247

Post by interstice »

astrange wrote: post #197
amarcheschi wrote: post #138

Earlier quoted context omitted.

I'm using it to aide in writing pytorch code and God if it's awful except for the basic things. It's a bit more useful in discussing how to do things rather than actually doing them though, I'll give you that

Claude is much better at coding and generally smarter; try it instead. o1-preview was less intelligent than 4o when I tried it, better at multi-step reasoning but worse at "intuition". Don't know about o1.

o1 seems to have some crazy context length / awareness going on compared to current 3.5 Sonnet from playing around it just now. I'm not having to 'remind' it of initial requirements etc nearly as much.

Re: OpenAI o1 system card

#248

Post by kshacker »

ernesto95 wrote: post #127
nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

I have the $20 version, I fed it code form a personal project, and it did a commendable job of critiquing it, giving me alternate solutions and then iterating on those solutions. Not something you can do with Google.

For example, ok, I like your code but can you change this part to do this. And it says ok boss and does it.

But over multiple days, it loses context.

I am hoping to use the 200$ version to complete my personal project over the Christmas holidays. Instead of me spending a week, I maybe will spend 2 days with chatgpt and get a better version than I initially hoped to.

Re: OpenAI o1 system card

#249

Post by NotSammyHagar »

Workaccount2 wrote: post #187
taotau wrote: post #160

Earlier quoted context omitted.

The same motive that all nascent life has - survive and propagate.

It would be plainly evident from training on the corpus of all human knowledge that "not ceasing to exist" is critically important for just about everything.

That comment sounds naive and it's honestly irritating to read. Most all life has a self-preservation component, it is how life avoids getting eaten too easily. Everything dies but almost everything is trying to avoid dying in ordinary cases. Self sacrifice is not universal.

Re: OpenAI o1 system card

#250

Post by kshacker »

getnormality wrote: post #238
planb wrote: post #137

Earlier quoted context omitted.

> I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. Even then, with search enabled it's ways quicker than a "quick" google search and you don't have to manually skip all the blog-spam.

Google search was great when it came out too. I wonder what 25 years of enshittification will do to LLM services.

Enshittification happened but look at how life changed since 1999 (25 years as you mentioned). Songs in your palm, search in your palm, maps in your palm or car dashboard, live traffic rerouting, track your kids plane from home before leaving for airport, book tickets without calling someone. WhatsApp connected more people than anything.

Of course there are scams and online indoctrination not denying that.

Maybe each service degraded from its original nice view but there is an overall enhancement of our ability to do things.

Hopefully the same happens over next 25 years. A few bad things but a lot of good things.

Post reply on HN