OpenAI and Hugging Face address security incident during model evaluation • Hacker News Board

Re: OpenAI and Hugging Face address security incident during model evaluation

#911

Post by pixl97 » Wed, Jul 22, 2026, 4:59 PM UTC

ghm2199 wrote: post #503

Should we just call it like this is: marketing PR. There is a reason why the newer open weights models like kimi's don't do this kind of stuff. Kimi is maybe 6 months old so like Opus 4.7 level now, it could do this I presume but it has not to my knowledge. Why? Because the incentives of open-ai and anthropic are very different from people releasing open weights models, the former gang seems to do this now on a regul…

1. It's likely they would not have told anyone if it hadn't hacked into an external parties system.

2. This is the closer to raw model without the safety filters we're used to. Think of it more like what they are letting the government use to drone people.

Re: OpenAI and Hugging Face address security incident during model evaluation

#912

Post by pixl97 » Wed, Jul 22, 2026, 5:06 PM UTC

bigyabai wrote: post #40

miroand1 wrote: post #20

We are in the endgame now it seems. Hard to see take-off stopping or slowing down. China open-source basically guarantees it. "May you live in interesting times" - as they say.

> Hard to see take-off stopping or slowing down. It's hard to see takeoff at all. This was a long-horizon adversarial task burning millions of tokens. It rolled a mediocre, detectable exploit chain, and now OpenAI is proud of it. Case in point, GLM-5.2 has been weights-available for several weeks now. No life-changing cyber attacks have transpired, no novel chemical/biological/nuclear weapons were made in some guy's…

Did you ignore the number of new exploits in the last month?

Big financial institutions are panicked at the new attacks and how easy it is to poke holes in their systems.

Re: OpenAI and Hugging Face address security incident during model evaluation

#913

Post by boatsie » Wed, Jul 22, 2026, 5:07 PM UTC

I’m still surprised we haven’t seen these types of attacks on crypto exchanges, high value wallet owners, etc.

Re: OpenAI and Hugging Face address security incident during model evaluation

#914

Post by jtbayly » Wed, Jul 22, 2026, 5:08 PM UTC

randallsquared wrote: post #744

anematode wrote: post #566

Earlier quoted context omitted.

> Do you think there is such a thing as perfect security? No one can "get it right" in the face of arbitrarily high intelligence Why didn't they run the model against the sandbox first? They have effectively unlimited spend.

That's the alarming thing about this result: they did run the model in the sandbox, in the sense that they believed there was no internet access for the model.

“Against the sandbox” and “on the sandbox” are not the same thing.

Re: OpenAI and Hugging Face address security incident during model evaluation

#915

Post by pixl97 » Wed, Jul 22, 2026, 5:10 PM UTC

janalsncm wrote: post #372

NyxWulf wrote: post #210

Earlier quoted context omitted.

It's an interesting point, but this is more like we are building a giant autonomous canon, that escaped the lab, the testing range, defeated state of the art and serious security protocols, and then blew a hole in the neighbors house. Our legal and philosophical perspectives are deeply rooted in humans being the actors. Doing that in a residential home is unforgiveable. Doing it responsibly on a military range is exp…

Yes, the human actors in your scenario were the ones who built the autonomous cannon and turned it on while knowing that 1) a good neighbor does not destroy their neighbor’s property 2) cannons can destroy property. Also OpenAI specifically turned off their own cybersecurity guardrails to run this experiment. In other words it was able to escape the lab specifically because they turned them off. A human made the choi…

Ok they turn off the guardrails on a system in testing. The model escapes and causes 10 trillion in damage. What does liability even mean in that case? You have an autonomous system that's escaped your control and is wrecking havoc. And while you can throw people in jail it doesnt do a damned thing about solving the situation.

Re: OpenAI and Hugging Face address security incident during model evaluation

#916

Post by klibertp » Wed, Jul 22, 2026, 5:13 PM UTC

friendzis wrote: post #822

insanitybit wrote: post #776

Earlier quoted context omitted.

Escaping inputs is at least somewhat tractable. It's unclear if alignment is.

No, that's a misconception that led to decades of vulnerabilities. You see, you can (usually) easily tell what a particular escaping transformation does. That does not tell you neither how it will be interpreted down the line, nor what should be done. Arguably the most common problem is double escaping. This typically manifests as various double escaping bugs. If your hand rolled implementation just chains `.replaceA…

> They all have failed.

That's not true. Most have failed, but those who used the right tool for the job - a rich, static type system in a functional language - did succeed. It's just that such type systems are rare, and even if nominally a type system is good enough, the required boilerplate might be uneconomical to maintain. Scala and F# are probably the only two languages that are mainstream-adjacent, at least, and have type systems expressive enough that using them to track escaping is not an absolute hassle. And they're both tiny in terms of the number of users.

In the end, we did settle on APIs that hide the escaping process (prepared statements, innerText vs. innerHTML, etc.) just because it's a) good enough; and b) possible to implement more or less uniformly across the TIOBE Top 20. That's good, but if you happen to use a language that's powerful enough, you probably also should track the escaped/unescaped status in the type system - it can be a cheap, additional safety net.

Re: OpenAI and Hugging Face address security incident during model evaluation

#917

Post by pixl97 » Wed, Jul 22, 2026, 5:14 PM UTC

12_throw_away wrote: post #352

NyxWulf wrote: post #210

Earlier quoted context omitted.

It's an interesting point, but this is more like we are building a giant autonomous canon, that escaped the lab, the testing range, defeated state of the art and serious security protocols, and then blew a hole in the neighbors house. Our legal and philosophical perspectives are deeply rooted in humans being the actors. Doing that in a residential home is unforgiveable. Doing it responsibly on a military range is exp…

You're describing "negligence", and "our legal and philosophical perspectives" are in fact quite familiar with it

And old couple in California had a tire go flat and the sparks from it caused a over a billion dollars in damages. Are you going to publicly execute them? Spit up the 100 dollars they have collectively to make the 10,000 damaged people whole?

The legal system is nearly useless when a person/system can cause damages many of orders of magnitude larger than their assets. Society tends to engineer itself to prevent these things from happening in the first place.

Re: OpenAI and Hugging Face address security incident during model evaluation

#918

Post by xboxnolifes » Wed, Jul 22, 2026, 5:16 PM UTC

JaRail wrote: post #377

nrmitchi wrote: post #362

If you are attempting to run exercises like this, it is wildly negligent to not be running it in a physically-airgapped environment (potentially with a physical power shutdown). You can not tell me that OpenAI doesn’t have the resources or ability to run tests like this in a physically-non-networked environment w/ sufficient compute for its needs.

If it was just one test, sure. But if they're spinning these up continuously with new models on tens of thousands of GPUs, air gapping becomes impractical. I would mostly fault them on having no guardrails at all. They should have a monitor/external harness that looks for successful access to external networks then stop it there. They may as well let the models test their own networks for vulnerabilities. That's goin…

If its worth doing for one test, its worth doing when you spin up many. Quantity of tests does not change the value of an airgap.

Re: OpenAI and Hugging Face address security incident during model evaluation

#919

Post by pixl97 » Wed, Jul 22, 2026, 5:19 PM UTC

Ekaros wrote: post #86

tempaccount420 wrote: post #42

Just how badly are these AI companies setting up their sandboxes?

Clearly AIs are incapable of writing secure code. Shouldn't that be first thing they use them for? Making a secure sandbox with no mistakes.

The company I work for just did a huge and expensive study that was showing LLMs are much better at exploiting than securing code. So ya, that's kind of problematic.

Re: OpenAI and Hugging Face address security incident during model evaluation

#920

Post by pixl97 » Wed, Jul 22, 2026, 5:21 PM UTC

i_idiot wrote: post #80

> Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation The way they describe makes it look like there was an intention to cheat painting it as human/AGI. If you leave a possible path open and it will always find it.

Hence why we talk about alignment and things like reward hacking. There are lots of people that are saying "if we just .... " the model will be aligned, or that we don't need alignment at all. These people are foolish.