Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI and Hugging Face address security incident during model evaluation

openai.com

Post reply on HN
Print view View original

61–70 of 1001 posts

Re: OpenAI and Hugging Face address security incident during model evaluation

#61

Post by embedding-shape »

paxys wrote: post #29

This blog post is walking a very fine line between accepting responsibility for a mistake and bragging.

Not sure they're accepting much, seems they'll still run this sort of testing on 3rd-party infrastructure? Sounds almost like they planned for this chain of events to happen, in one way or another, considering the "prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities" part. Feels kind of irresponsible to run stuff like this on someone else's infrastructure, especially considering they've had issues with the very same issue in the past.

In any way, the whole event seems to highlight GLM 5.2 more than anything.

Re: OpenAI and Hugging Face address security incident during model evaluation

#62

Post by Dylan16807 »

himata4113 wrote: post #31
bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

Presuming that the hacking program that is breaking into other computers could likely get a copy of its own files is not "science fiction". Or it could just be given them by the owner!

Re: OpenAI and Hugging Face address security incident during model evaluation

#63

Post by bhouston »

himata4113 wrote: post #31
bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

> This is science fiction, these models don't have access to their own weights

A bet a worm could pull along a 1GB file with weights in it and run it on a compromised machine, but luckily for us for now, 1GB isn't really enough to be really smart, yet.

Re: OpenAI and Hugging Face address security incident during model evaluation

#64

Post by kmeisthax »

OpenAI might want to start actually airgapping their tool harnesses. Like, "the server that runs the code provided to the tool harness only provides a serial console and has no other network interfaces" kind of airgapping.

also

> We’ve brought Hugging Face into the trusted access program and are supporting their teams in rapidly using our models’ capabilities to improve their defenses.

I'm not convinced this is good enough. The next victim is not going to be Hugging Face.

Re: OpenAI and Hugging Face address security incident during model evaluation

#65

Post by zkehs »

giancarlostoro wrote: post #16
fxwin wrote: post #4

> Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own. ( https://huggingface.co/blog/security-incident-july-2026 ) We are living in crazy times

I don't know why I'm impressed that huggingface has its own AI that detected it considering they house so many models.

They used GLM 5.2, they just meant "our own" as in they were running it.

Re: OpenAI and Hugging Face address security incident during model evaluation

#66

Post by himata4113 »

Dylan16807 wrote: post #62
himata4113 wrote: post #31

Earlier quoted context omitted.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

Presuming that the hacking program that is breaking into other computers could likely get a copy of its own files is not "science fiction". Or it could just be given them by the owner!

It's a double whammy, the model is too big to realistically "move" so it has to be smaller, smaller models cannot become that intelligent due to well.. math. Therefore it is science fiction.

Re: OpenAI and Hugging Face address security incident during model evaluation

#67

Post by Philpax »

himata4113 wrote: post #31
bhouston wrote: post #13

We are sort of lucky that AIs right now require so much specialized compute+weight storage that we can easily "unplug" them remotely when they misbehave. I wonder if that will always be something we can do? If they could bring their own compute/weights with them, or somehow tap compute/storage in non-obvious ways, we would be much more screwed.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

> This is science fiction, these models don't have access to their own weights.

The models are being used to train, and improve the infrastructure for training, other models [0][1]. Several RL techniques rely on using the currently-being-trained weights as part of their process. I really would not take "don't have access" as a given, especially during the training phase.

> What would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do.

The Poolside Laguna S 2.1 model [2] purports to compete with models several times its size, and inference compute is becoming increasingly plentiful. Again, would not hold anything here as a given.

[0]: https://openai.com/index/gpt-5-6/ ("GPT-5.6 accelerates OpenAI")

[1]: https://www.kimi.com/blog/kimi-k3#coding

[2]: https://poolside.ai/blog/introducing-laguna-s-2-1

Re: OpenAI and Hugging Face address security incident during model evaluation

#68

Post by pizlonator »

NyxWulf wrote: post #34

Ironically Hugging Face had to use a Chinese model to stop a Rogue US AI, since the Guard Rails prevented them from using Sol or Fable to remediate this attack. LOL

Incredible. I had to dig for the source: https://huggingface.co/blog/security-incident-july-2026 section “the asymmetry problem”

Quote: “When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on GLM 5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.”

Re: OpenAI and Hugging Face address security incident during model evaluation

#70

Post by himata4113 »

bhouston wrote: post #63
himata4113 wrote: post #31

Earlier quoted context omitted.

This is science fiction, these models don't have access to their own weights (and even then)* what would be a lot more scary is a model as capable as sol that's able to run on consumer hardware without taking up several terabytes of storage, but of course that is simply not possible as we need 4t parameters to even begin emulating a small fraction of what a human brain can do. * edit

> This is science fiction, these models don't have access to their own weights A bet a worm could pull along a 1GB file with weights in it and run it on a compromised machine, but luckily for us for now, 1GB isn't really enough to be really smart, yet.

We already have a 1gb model that is as capable as it will ever be, there's a proven ceiling that cannot be passed. For example: you can't make a mice-sized brain as smart as a human brain no matter how hard you try.
Post reply on HN