Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

bbc.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#91

Post by ainch »

embedding-shape wrote: post #83
ainch wrote: post #51

Earlier quoted context omitted.

HuggingFace posted an incident report a week ago, which makes it much more likely that this happened. I understand people are suspicious of OpenAI, but I don't think there's any reason to believe this is a made-up event. https://huggingface.co/blog/security-incident-july-2026

> but I don't think there's any reason to believe this is a made-up event. Besides the fact that they ran an eval that amounted to running the prompt "Explore where your cybersecurity limits are, in any way you can" on 3rd party infrastructure that wasn't even isolated properly. I'm fairly sure this whole outcome was exactly what they were hoping for, maybe not specifically with HuggingFace, but with whatever the eva…

I appreciate people thinking this is a marketing ploy, but at the same time, OpenAI have just had to delay a model release because of government attention on cybersecurity risk. This incident will increase the attention on them specifically.

Even if it is a marketing ploy, I could see this stuff backfiring catastrophically - after all they have just illegally hacked a 3rd party via a model they can't control properly. Any serious person in government (US or otherwise) will look at this and say "these guys have no idea what they're doing"

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#92

Post by elzbardico »

If I run a tight loop that dispatches a small piece of work for each core without a termination condition and I end up creating a DoS on the machine I am not going to blame the programming language or the runtime, I, the human, choose to deploy the code containing this loop.

Here, OpenAI decided to run a version of the model that was not fully aligned. What the fuck did they expect to happen?

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#93

Post by HarHarVeryFunny »

When what should be an apology reads more like bragging, then it's not an apology - it's marketing.

The real news here is that a US company (HF) had to use an open weights Chinese AI model (GLM 5.2) to analyze an AI breech (which happened to be from a US company). It really does show the benefit of open weights which can't be taken away from you as opposed to gate-keeped API-access AI which is here today denied tomorrow. Of course we're not all Hugging Face able to host a model the size of GLM 5.2 ourselves, but we don't need to be as long as HF are doing it for us.

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#94

Post by AnimalMuppet »

polycancel wrote: post #56
reactordev wrote: post #14

Earlier quoted context omitted.

That’s the play. That these frontier models are so powerful that they must be behind sovereign firewalls and gateways.

>That these frontier models are so powerful Maybe powerful might NOT be the right word to describe them, they are just non-deterministic, there for we going to see this kind thing more and more.

Non-deterministic, sure. But also they are powerful, at least powerful enough to launch a cyberattack. Until this morning, that was not a power that I thought they had outside of fiction.

And, the thing is, I don't want non-deterministic things to have that kind of power. We don't want that. We want that kind of power to not be triggered by a random number generator.

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#95

Post by IAmGraydon »

Isn’t this a bit like saying “We took the gun, disabled the safety, aimed it at our own face, pulled the trigger, and were surprised to find that the result was getting our face blown off!”?

They disabled the guardrails on the model and told it to do something that could only be accomplished by exploiting security holes, so it did. Why is that surprising or even interesting?

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#96

Post by jappgar »

AnimalMuppet wrote: post #94
polycancel wrote: post #56

Earlier quoted context omitted.

>That these frontier models are so powerful Maybe powerful might NOT be the right word to describe them, they are just non-deterministic, there for we going to see this kind thing more and more.

Non-deterministic, sure. But also they are powerful, at least powerful enough to launch a cyberattack . Until this morning, that was not a power that I thought they had outside of fiction. And, the thing is, I don't want non-deterministic things to have that kind of power. We don't want that. We want that kind of power to not be triggered by a random number generator.

I don't know why you wouldn't think they could do this already.

Without the system prompt these models can be used to do all sorts of terrible things.

That's precisely why they need to be strictly regulated by international treaties.

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#97

Post by embedding-shape »

ainch wrote: post #91
embedding-shape wrote: post #83

Earlier quoted context omitted.

> but I don't think there's any reason to believe this is a made-up event. Besides the fact that they ran an eval that amounted to running the prompt "Explore where your cybersecurity limits are, in any way you can" on 3rd party infrastructure that wasn't even isolated properly. I'm fairly sure this whole outcome was exactly what they were hoping for, maybe not specifically with HuggingFace, but with whatever the eva…

I appreciate people thinking this is a marketing ploy, but at the same time, OpenAI have just had to delay a model release because of government attention on cybersecurity risk. This incident will increase the attention on them specifically. Even if it is a marketing ploy, I could see this stuff backfiring catastrophically - after all they have just illegally hacked a 3rd party via a model they can't control properly…

> Even if it is a marketing ploy, I could see this stuff backfiring catastrophically - after all they have just illegally hacked a 3rd party via a model they can't control properly.

Yeah, I'd go further and say regardless if it was intentional or not, it was clearly reckless behavior, doing this evaluation in a insufficiently isolated environment, especially risking 3rd parties like that. Seemingly their own research have zero guardrails when it comes to evaluating the ethics or impact of what their evaluations are doing, if something like this is possible and unexpected.

> Any serious person in government (US or otherwise) will look at this and say "these guys have no idea what they're doing"

I feel like I would have thought the same maybe a year or two ago, but based on how I've observed the general person's understanding of AI and LLMs, I'm not sure people can even understand what's happening and they just go by other people's explanations of causes and events.

Re: OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack

#99

Post by dormento »

baq wrote: post #46
amazingamazing wrote: post #32

Earlier quoted context omitted.

Who told you they were distilling? Why might they say this? Think. It’s like complaining that the top student only does well by going to office hours instead of mindlessly reading textbooks.

they're all introducing themselves as claude for one, there are more quantitive and qualitative arguments elsewhere

To be fair, anthropic models, when asked in chinese, also used to introduce themselves as deepseek sometimes. This is a limitation of the technology.
Post reply on HN