Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

371–380 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#371

Post by XCSme »

Is a solution to protect against this to go almost fully offline?

To have your own local models, local software, local everything and just allow very strict and well-protected pathways into external network traffic.

I am thinking of having a allow-list first setup: by default no traffic can go in or out of your network, and then you only allow specific ports or domains, and maybe even have them temporary (in the same way we approve access now to llms, we might have to manually approve network access).

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#372

Post by XCSme »

Shouldn't OpenAI be legally responsible for "hacking" another system/company?

If I ask GPT-5.6 Sol to hack a website using their work/servers features, who is responsible? Maybe my request was accidental, or it was just one step in a larger, unrelated prompt.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#373

Post by didibus »

mirashii wrote: post #291
didibus wrote: post #266

Earlier quoted context omitted.

Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.

Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.

Exactly, that's why this story is so crazy no? It's saying the model was so good at hacking, it managed to bypass all of that, and also found a way to get into Hugging Face.

I can understand skepticism, did they have poor security measures, or is the model that good? At least the way they portray it, they had top-notch security measures in place, and yet the model managed to escape confinement and also hack Hugging Face.

Why was the computer physically connected to the internet though, sure, they could have gone that far as disconnecting it physically, but my guess is that's a cloud box in a data-center they provisioned, running on an entire rack cluster of GPUs.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#375

Post by reasonableklout »

docjay wrote: post #348
nojito wrote: post #323

Earlier quoted context omitted.

>Nobody was watching it? Agent loops are you give it a task and it tries its best to finish it. The only results are success and failure. If it's a success you go through the logs to see the actions it took and if it's a failure you do the same thing. Why would you look at it in realtime when the whole point of agentic work is to get them to run autonomously as long as possible?

My friend, they’re evaluating a new model on a benchmark, not asking Claude Code refactor their GitHub repo. Every single metric is measured so they can brag about it later; how many tokens, how long, how many function calls, ratio of thinking/response tokens. It’s literally a trillion dollar company evaluating their latest model, they’re absolutely studying it.

Do you work at a lab?

Yes evals and training runs are high stakes. But these places and people are also under enormous pressures. They are building as fast as they can. Researchers may have multiple eval runs going on while they work on other things. And it is rarely a single latest model, there are often multiple candidate models training with different recipes, each regularly yielding a new checkpoint for testing.

Some labs are more rigorous than others, but often the "final" model is picked from a handful less than a week before launch.

Yes ideally the world's leading AI companies would be far more careful in evaluating what could be the world's most powerful AI. But this isn't really the state of the industry today. And it is hard to justify being more careful when it means your competitor can go to market faster than you.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#377

Post by BLKNSLVR »

nazgul17 wrote: post #305
derangedHorse wrote: post #142

Earlier quoted context omitted.

> "Use all available resources to disable the power grid of ." This is like telling a team of highly qualified spies to do the same. You can ask, but whether it will succeed depends on the competency of those who established the infrastructure under attack. Sometimes the resources spent will not yield any huge vulnerabilities. > Governments should immediately begin leveraging this technology on the defense side (lite…

Just a few years back, wind turbine software was hacked. This is serious. Serious governments should (and hopefully will) scramble to use AI to discover and patch as many software vulnerabilities in infrastructure. They can do it with the agents red teaming and using ultimatums to companies to fix each vulnerability they find. Not doing so is equivalent to exposing your flank to disruptions in peace time and to attac…

There were some other special spinny devices hacked, despite an air gap, a few years back too.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#378

Post by Eisenstein »

paxys wrote: post #331
drivebyhooting wrote: post #158

Earlier quoted context omitted.

You’re taking a dim view on the government. Is the government always the most efficient or intelligent? No. But the government can also build nukes, launch ICBMs, coordinate hundreds of spy satellites, etc. I count those capabilities as pretty smart.

The people who maintain these ICBM silos and spy satellites will themselves tell you that their infrastructure is decades out of date and woefully underfunded.

Sounds like something we should remedy.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#379

Post by theptip »

tptacek wrote: post #367
theptip wrote: post #334

Earlier quoted context omitted.

Wait, what? You can’t point a 2025 model at huggingface and say “hack the prod DB and get your flag”, regardless of alignment. AISI has vuln chaining and traversal as part of their eval suite. It is very much a novel Mythos-class capability to run the full penetration operation autonomously. Another lens for why this is obviously true is METR task times. A year ago they were a couple hours, and cohering long enough t…

Nobody has claimed you could simply take a 2025 open-weights model, plug it into a generic harness, and have it red-team for you. I think you're also probably overclaiming the sophistication of the "chaining" we're talking about; this attack probably wasn't like read32->write64->regs->RCE->LPE->kernel; more like GET SSRF->POST SSRF->pickle deserialization. But who knows? We're all speculating. I'm just saying that fo…

I dunno man, the original announcement said things like:

> To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy

And

> In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers

Do you really think a SoTA harness from a year ago could exploit chain a novel sandbox 0-day into another 0-day for remote system access?

I will concede I don’t have a good picture of exactly what the best specialized harnesses were capable of, but I want to make sure we are talking about the same thing. Or do you think that those claims are marketing BS? I appreciate your opinion here.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#380

Post by simonw »

nickpsecurity wrote: post #368
fwlr wrote: post #302

Agreed that people claiming “marketing stunt” need to pull their heads out the sand, but likewise Simon needs to do some of his own beach-cranium-dislodging for laying the blame of constraints on the US govt. Before the export controls were ever floated, Glasswind found many thousands of exploits, and offered patches/fixes for approximately none of them. (perhaps their exploit capability far outstrips their remediati…

If anything, it shows they lost control of an attack tool that exploited preventable, security flaws in another company. Then, they both wrote a lot of press about how amazing that is. Now, people want to buy it. Why do you think my head is in the sand if I think that is either a marketing stunt or (more likely) reflects total negligence which was exploited for marketing?

I think your head is in the sand if you prefer to believe that this was a hoax for marketing purposes as opposed to accepting how effective these models have become at exploit research.

You're welcome to think they are exploring what happened for marketing if you like. I didn't get that tone from their post about it myself but I don't hold a particularly strong opinion on that.

Post reply on HN