Timeline of the OpenAI accidental attack against Hugging Face • Hacker News Board

Re: Timeline of the OpenAI accidental attack against Hugging Face

#191

Post by [deleted] » Sat, Aug 08, 2026, 5:26 PM UTC

skydhash wrote: post #157

simonw wrote: post #137

Earlier quoted context omitted.

If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.

It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions. We have even observed that the most effective LLM usage is when paired with an…

[deleted]

Re: Timeline of the OpenAI accidental attack against Hugging Face

#192

Post by simoncion » Sat, Aug 08, 2026, 5:32 PM UTC

tsimionescu wrote: post #142

simoncion wrote: post #113

Earlier quoted context omitted.

> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ens…

> "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?". I feel this completely misunderstands the problem, and the vast gulf between an LLM and a CPU. First and most importantly, the set of behaviors of a CPU is extremely constrained, and we have a very simple model for which behaviors are safe and which are not. Writing to addresses between X and Y, executing certain in…

> Secondly, CPUs can absolutely be used to hack other people.

This is more correctly phrased as "Every general-purpose computer can be run any arbitrary program, assuming it has the storage required to load that program.". Despite that fact, we've managed to learn how to write programs that run on those computers that fail to give attackers who have control of the inputs to those programs control of the instructions those programs feed to the CPU. This part of your argument strengthens my point.

> First and most importantly, the set of behaviors of a CPU is extremely constrained...

The techniques we use to prevent data our programs process from altering the instructions we send along to our CPUs work regardless of instruction set complexity. This objection of yours is irrelevant.

A CPU does not know who authored the next instruction it is to run. A CPU only knows to execute instructions handed to it. Despite the fact that CPUs are dumb as bricks and have zero understanding of where their instructions come from, we've -somehow- managed to learn how to build software that operates on untrusted data without relinquishing control of the CPU's instruction stream to attackers.

The LLM providers ignored the most basic lesson of the last ~fifty years of secure software design. This was economically a very smart thing to do, but an absolute catastrophe for the health of computing.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#193

Post by KingOfCoders » Sat, Aug 08, 2026, 5:33 PM UTC

ejpir wrote: post #183

KingOfCoders wrote: post #164

Earlier quoted context omitted.

[edit] I've now watched the video on the idea that your write-up was misleading. BUT the video is much worse. For two months with highly dangerous agents agents were hacking a service and none of the researchers watched (drank coffee for 2 months, didn't say). THEN they found the hack, removed the message board. AND the agents found another way to create a message board, on the same service, and the researchers again…

Claude Code is not the same as the models they train and use internally, for both OAI and Ant. Without all the guard rails it behaves different, they specifically mentioned that they reduced the refusals for the training purposes. Also the rewards for finding the solution were set higher.

An agent idling and then acting on it's own to hack HF is has nothing to do with guard rails.

Someone had to give the agent some instructions, like "hack X", "Find exploit for Y" or "Do whatever havoc you can think of" - either way the agents didn't not act on their own. They might hack HF on their own, today Claude decided to play sound through the sound pipeline I instructed it to build and measure it to see if it works, but it didn't install the sound pipeline because it hasn't had anything better to do but because I instructed it that way.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#194

Post by MostlyStable » Sat, Aug 08, 2026, 5:33 PM UTC

simonw wrote: post #93

I think one of the most interesting details here might be tucked away in that first bulletin point: > May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.) The…

Yes, the message boards and collaborative hacking occurring during training runs was BY FAR the biggest bombshell revealed, and OpenAI doesn't even seem to realize it. The fact that they continued the training runs, with those rewarded behaviors included, and didn't wind back training to before hand, shows that they fundamentally do not understand alignment and safety (somewhat interestingly, their previous head of safety resigned shortly after OpenAI found about the message boards). I agree that, with that information, it is completely unsurprising that they hacked HuggingFace.....but that is also the Star Wars "You understand how that's worse, right?" meme.

I am flabbergasted at the complete lack of regard for alignment demonstrated here.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#195

Post by icebxrg » Sat, Aug 08, 2026, 5:34 PM UTC

RGS1811 wrote: post #46

Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…

"Car accidents occur therefore we shouldn't have cars" isn't very compelling.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#196

Post by singingtoday » Sat, Aug 08, 2026, 5:35 PM UTC

qsera wrote: post #182

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

> instead just call defeat and say “I’m not sure how to proceed next”. Because that is fundamentally impossible given how they work... The thing does not even know when it succeeds or fails. Actually the thing does not "know" at all... All it can does is to show some limited textual behavior that matches with "knowing"..

You can get near this point with scaffolding. Keep in mind, LLMs are next word predictors at their root. More abstractly, they capture and replay likely human intelligence by way of written language. Tokens.

With that concept in mind, it's clear how they can be made to "give up".

Re: Timeline of the OpenAI accidental attack against Hugging Face

#197

Post by simoncion » Sat, Aug 08, 2026, 5:36 PM UTC

Covenant0028 wrote: post #138

simoncion wrote: post #113

Earlier quoted context omitted.

> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ens…

The entire economic premise and value case of LLMs rests on the idea that instructions need not be provided in advance, and that the model can "reason" based on evidence and "decide" what to do next. Even if it were technically possible to separate instructions from code and ensure that the LLM only followed those, it would require someone to specify the instructions in advance (ie a program), at which point the LLM…

> ...it would require someone to specify the instructions in advance (ie a program)...

What do you call "A user typing instructions into the Python or Ruby interactive CLI."? How is that a meaningfully different method of computer instruction than "A user typing instructions into the Claude or Codex interactive CLI."?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#198

Post by gizajob » Sat, Aug 08, 2026, 5:37 PM UTC

KingOfCoders wrote: post #187

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking the unsecure service, no tripwire, researchers find the hack days/weeks/months later, fix it, but don't secure the sandbox and the service was hacked a second time, again without being monitored by security researchers. Then security researchers create a black…

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either.

Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared.

OpenAI hacking HuggingFace and calling it an accident is just way too convenient and fishy. This ultimately proves one thing: it wasn't sandboxed.

Don't believe the hype.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#199

Post by singingtoday » Sat, Aug 08, 2026, 5:37 PM UTC

rolls-reus wrote: post #63

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”. that might end up like the older gemini models which frequently gave up and called itself a failure.

Gemini still gives up too easily

Re: Timeline of the OpenAI accidental attack against Hugging Face

#200

Post by qsera » Sat, Aug 08, 2026, 5:38 PM UTC

singingtoday wrote: post #196

qsera wrote: post #182

Earlier quoted context omitted.

> instead just call defeat and say “I’m not sure how to proceed next”. Because that is fundamentally impossible given how they work... The thing does not even know when it succeeds or fails. Actually the thing does not "know" at all... All it can does is to show some limited textual behavior that matches with "knowing"..

You can get near this point with scaffolding. Keep in mind, LLMs are next word predictors at their root. More abstractly, they capture and replay likely human intelligence by way of written language. Tokens. With that concept in mind, it's clear how they can be made to "give up".

>With that concept in mind, it's clear how they can be made to "give up".

They can, but they need to be trained specifically on that behavior. They can be trained specifically to not generate textual description of things that look like hacking. But it is going to cost $$$, and as we currently see, most people don't care...