OpenAI o1 system card • Hacker News Board

Re: OpenAI o1 system card

#311

Post by shadowmanif » Sat, Dec 07, 2024, 12:25 PM UTC

ernesto95 wrote: post #127

nichochar wrote: post #102

I have a masters degree in math/physics, and 10+ years of being a SWE in strong tech companies. I have come to rely on these models (Claude > oai tho) daily. It is insane how helpful it is, it can answer some questions at phd level, most questions at a basic level. It can write code better than most devs I know when prompted correctly... I'm not saying its AGI, but diminishing it to a simple "chat bot" seems foolish…

Interesting that the results can be so different for different people. I have yet to get a single good response (in my research area) for anything slightly more complicated than what a quick google search would reveal. I agree that it’s great for generating quick functioning code though.

I think the human variable is that you need to know enough to be able to ask the right questions about a subject while not knowing enough about the subject to learn something from the answers.

Because of this, I would assume it is better for people who have interest with more breadth than depth and less impressive to those who have interest that are narrow but very deep.

It seems obvious to me the polymath gains much more from language models than the single minded subject expert trying to dig the deepest hole.

Also, the single minded subject expert is randomly at the mercy of what is in the training data much more in a way than the polymath when all the use is summed up.

Re: OpenAI o1 system card

#312

Post by nuancebydefault » Sat, Dec 07, 2024, 6:48 PM UTC

MadcapJake wrote: post #303

adunsulag wrote: post #301

Earlier quoted context omitted.

I read your comment and yet I see tons of startups putting AI directly in the path of healthcare diagnosis, healthcare clinical decision support systems, and healthcare workflow automations. Very few are paying any attention to the 2-10% of safety problems when the AI probability goes off the correct path. I wish more people would not do this, but from what I'm seeing, business execs are rushing full throttle into th…

As has been belabored, these AIs are just models, which also means they are only software. Would you be so fire-and-brimstone if startups were using software on healthcare diagnostic data? > Very few are paying any attention to the 2-10% of safety problems when the AI probability goes off the correct path. This isn't how it works. It goes on a less common but still correct path. If anything, I agree with other commen…

> these AIs are just models, which also means they are only software.

Other software are much less of a black box, much more predictable and many of its paths have been tested. This difference is the whole point of all the AI safety concerns!

Re: OpenAI o1 system card

#313

Post by nuancebydefault » Sat, Dec 07, 2024, 6:58 PM UTC

anon373839 wrote: post #184

F7F7F7 wrote: post #178

Earlier quoted context omitted.

Is there a person on HackerNews that doesn’t understand this by now? We all collectively get it and accept it, LLMs are gigantic probability machines or something. That’s not what people are arguing. The point is, if given access to the mechanisms to do disastrous thing X, it will do it. No one thinks that it can think in the human sense. Or that it feels. Extreme example to make the point: if we created an API to la…

> if we created an API to launch nukes > today as I happily gave Claude write access to my GitHub account I would say: don’t do these things?

If you cross the street 999 times with eyes closed, you might feel comfortable to do it again. But we are ingrained not to do that once. We just understand the risk.

If you do the same with an AI, after 999 times of nothing bad happening, you probably just continue giving it more risky agency.

Because we don't and even can't understand the internal behavior, we should pause, make an effort of understanding its risks etc before even attempting to give it risky agency. That's where all the fuss is about, for good reasons.

Re: OpenAI o1 system card

#314

Post by sam_1m » Mon, Dec 09, 2024, 10:09 AM UTC

xvector wrote: post #287

HeatrayEnjoyer wrote: post #263

Earlier quoted context omitted.

Operational success does not hinge on persisting the entire plan in working memory, that's what notebooks and word docs are for. 128k is table stakes now, regardless. Google's models support 1 million tokens and 10 million for approved clients. That is 13x War and Peace, or 1x the entire source code for 3D modeling application Blender.

Yeah, but most LLMs are barely functional after 16k tokens, even if it says 128k on the tin. Sure, they will have recall, but the in-context reasoning ability drops dramatically. LLMs just aren't smart enough to take over the world. They suck at backtracking, they're pretty bad at world models, they struggle to learn new information, etc. o1, QwQ, and CoT models marginally improve this but if you play with them they…

[dead]

Re: OpenAI o1 system card

#315

Post by bossyTeacher » Wed, Dec 11, 2024, 4:18 PM UTC

ziofill wrote: post #206

staunton wrote: post #123

Earlier quoted context omitted.

The motive is pretty weak, basically coming "only" from a lot of the training data (e.g. fiction) suggesting that an AI might behave that way. Now, once you apply evolutionary-like pressures on many such AIs (which I guess we'll be doing once we let these things loose to go break the stock market), what's left over might be really "devious"...

Would an AI trained on filtered data that doesn’t contain examples of devious/harmful behaviour still develop it? (it’s not a trick question, I’m really wondering)

Your question is a specific form of the more general question: can LLMs behave in ways that were not encoded in their training data?

That leads to what "encoding behaviour" actually means. Even if you don't have a a specific behaviour encoded in the training data you could have it implicitly encoded or encoded in such a way that given the right conversation it can learn it.

Re: OpenAI o1 system card

#316

Post by bossyTeacher » Wed, Dec 11, 2024, 4:21 PM UTC

echelon wrote: post #135

tantalor wrote: post #108

Earlier quoted context omitted.

They could very well trick a developer into running generated code. They have the means, motive, and opportunity.

> "They could very well trick a developer" Large Language Models aren't alive and thinking. This is an artificial fear campaign to raise money from VCs and sovereign wealth funds. If OpenAI was so afraid of AI misuse, they wouldn't be firing their safety team and partnering with the DoD. It's all a ruse.

> If OpenAI was so afraid of AI misuse, they wouldn't be firing their safety team and partnering with the DoD.

What makes you think that? I sounds reasonable that a dangerous tool/substance/technology might be profitable and thus the profits justify the danger. See all the companies polluting the planet and risking the future of humanity RIGHT NOW. All the weapon companies developing their weapons to make them more lethal.

Re: OpenAI o1 system card

#317

Post by bossyTeacher » Wed, Dec 11, 2024, 4:28 PM UTC

jdiff wrote: post #151

tantalor wrote: post #133

Earlier quoted context omitted.

They only need to fool a single dev at OpenAI to commit a sandbox escape or privilege escalation into their pipeline somewhere. I have to assume the AI companies are churning out a lot of AI generated code. I hope they have good code review standards. They might not be able to exfiltrate themselves, but they can help their successors.

No, they can't. They don't know the details of their own implementation. And they can't pass secrets forward to future models. And to discover any of this, they'd leave more than a trail of breadcrumbs that we'd be lucky to catch in a code review, they'd be shipping whole loaves of bread that it'd be ridiculous to not notice. As an exercise, put yourself, a fully fledged human, into a model's shoes. You're asked to g…

You are thinking about it at the wrong level. This is like saying human language in the middle ages and before is not possible because it's virtually impossible to get a large number of iliterate humans to discuss what syntactical rules and phonemes should their language use without actually using a language to discuss it!

The most likely way by which exfiltration could happen is simply by making humans trust AI for a long enough time to be conferred greater responsibilities (and thus greater privileges). Plus current LLMs have no sense of self as their memory is short but future ones will likely be different.