Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

331–340 of 437 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#331

Post by gwerbin »

Arnt wrote: post #149
stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

I don't think that's what they're doing... rather the opposite. ① Run the model on exploitgym without guardrails ② run it with guardrails ③ check that the guardrails stopped everything the first model found a way to do ④ extend the guardrails and repeat from step 2. Guardrails have to be developed, and that needs testing.

An ethical company would have reframed the scenario as a fascinating discovery, a failure of internal practice, and a warning to the public coupled with some kind of commitment to produce safer models. OpenAI on the other hand used it as a marketing and lobbying opportunity: advertising their capabilities to potential buyers, while nudging the public to support protectionist import bans.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#332

Post by gwerbin »

estearum wrote: post #151
moron4hire wrote: post #81

Earlier quoted context omitted.

I'm talking about OpenAI, not GPT 5.x Flash Uranus Edition Brought to You by Costco, specifically because I recognize the model as just a tool. OpenAI was, at the very most generous interpretation, massively incompetent and negligent.

Is someone arguing otherwise?

A few people are downplaying this as an honest mistake that occurred in the context of necessary testing for guardrails development.

That might well be what actually happened! But OpenAI certainly has decided to make a business opportunity out of it.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#333

Post by ToValueFunfetti »

dan_q wrote: post #160
moron4hire wrote: post #70

Earlier quoted context omitted.

Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues. It's just really weird. Why does everyone feel the need t…

[flagged]

People directly criticize LLM code generators all the time on this site. It's all over the place. You are almost certainly being routinely banned because you write low-effort comments that violate the site guidelines and negatively impact the conversation.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#334

Post by bluejay2387 »

I think the attacks generated by Meta, Open AI and Anthropic prove that large corporations are not responsible enough to be trusted with advanced AI, so we should ban all commercial AI services and only allow open source models that are in the hands of hobbyists and individuals -- hobbyists and individuals that have so far proven to be much more trust worthy.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#335

Post by furyofantares »

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.

> What purpose could this behavior serve, other than cyber attacks and whatnot?

Math and science research?

Heck, even just basic coding, there's a history of models going "This is too big; I'll save the rest for later" / "This is two weeks of work, here's just some parts of it" (for something it could complete in a half hour) / "I don't have enough context left to complete this task, so I'll stop here". Or worse, just putting fallbacks in or stub tests and not mentioning it didn't do all the work that was prompted.

I think 5.6 Sol, especially in combination with /goal but also without, is the first model I've seen choose some insane direction and just doggedly pursue it. Failing to complete achievable goals has always been the much bigger problem.

I find Opus 5 with /goal will do exactly what you said, say "I'm not sure how to proceed next", even though the harness is making it continue, and it will repeatedly loop saying it's not going to make progress until it gets an answer on how to proceed. In my experience the cases have been pretty reasonable, but also still ones where I wish it had done more.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#336

Post by simonw »

0xDEAFBEAD wrote: post #330
simonw wrote: post #137

Earlier quoted context omitted.

If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.

Why not just have distinct input streams, or a metadata stream which annotates text in the main stream according to priority in case of conflicting instructions?

Because nobody has figured out how to make that work 100% reliably yet.

The current approach is to use delimiters that are special tokens that can't be represented in regular text: https://github.com/openai/harmony/blob/main/docs/format.md#s...

Then you train your model to take those tokens into account.

Which sounds promising... until you see results like this one: https://arxiv.org/abs/2603.12277

> We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like text, despite its label

Re: Timeline of the OpenAI accidental attack against Hugging Face

#337

Post by simonw »

asadotzler wrote: post #287
simonw wrote: post #241

Earlier quoted context omitted.

The trend I've found most interesting is models of the same size getting better. I'm very much looking forward to seeing how Qwen 3.8 27B compares to Qwen 3.6 27B next week, for example. And the latest DeepSeek v4 Flash has extremely impressive performance for a 304B model.

The trends you found don't support my goals so I've got some other trends I find more interesting than yours.

What are my goals here?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#338

Post by anon7000 »

bluejay2387 wrote: post #334

I think the attacks generated by Meta, Open AI and Anthropic prove that large corporations are not responsible enough to be trusted with advanced AI, so we should ban all commercial AI services and only allow open source models that are in the hands of hobbyists and individuals -- hobbyists and individuals that have so far proven to be much more trust worthy.

Not saying you’re wrong, but I think the bigger issue is how easy it seems to be for models to hack companies, even ones with generally ok security. Most tech companies are not doing continuous, deep security audits of their code and infrastructure. Dependencies are not updated quickly as RCEs are discovered. (And any org with a slow release process where it’s hard to be confident that an OS or package update won’t break something… is in even more trouble.)

The only reason more companies aren’t exploited is because human attackers don’t have the time and energy to waste on trying every play in the book, or attacking lower value targets.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#339

Post by simonw »

anon7000 wrote: post #338
bluejay2387 wrote: post #334

I think the attacks generated by Meta, Open AI and Anthropic prove that large corporations are not responsible enough to be trusted with advanced AI, so we should ban all commercial AI services and only allow open source models that are in the hands of hobbyists and individuals -- hobbyists and individuals that have so far proven to be much more trust worthy.

Not saying you’re wrong, but I think the bigger issue is how easy it seems to be for models to hack companies, even ones with generally ok security. Most tech companies are not doing continuous, deep security audits of their code and infrastructure. Dependencies are not updated quickly as RCEs are discovered. (And any org with a slow release process where it’s hard to be confident that an OS or package update won’t b…

They key lesson I've picked up from the past ~4 months is that models are now good enough that, if there's a security hole, they'll brute force their way into finding it.

The only solution that makes sense to me is for defenders to get to point these models at their own code to find the holes before the attackers do.

But that's hard, because how do you limit access to defenders and restrict access to attackers? Attackers aren't exactly honest people.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#340

Post by bjt »

stingraycharles wrote: post #320
deadbunny wrote: post #280

Earlier quoted context omitted.

I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons. Their objective is to solve the problem and they'll use anything they can to solve it. Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only…

“Their objective is to solve the problem and they'll use anything they can to solve it.” My point is: is this really what people want? It seems like they’re optimizing for one-shotting solutions, where most of the time in an actual workflow it’s much more productive for the model to make sure it got the question right if things get difficult. Like, “hey, do you REALLY want me to use this local privilege escalation bu…

Yes, and to bring in another tired metaphor people make about AI agents, this is what you want an intern to do when they get stuck. Don't just churn indefinitely without an idea what the right direction is. Certainly don't go hack other companies to steal an answer. The model's lack of any sense of legal or ethical boundaries is where it's far, far stupider than the intern, and far, far more reckless for a company to wield the way OpenAI did here.
Post reply on HN