Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

431–440 of 440 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#431

Post by InvidFlower »

tolleyw wrote: post #379
simonw wrote: post #304

Earlier quoted context omitted.

Because it's a crime. Committing crimes is a bad look for companies, especially given the amount of scrutiny they are under. Would you deliberately commit computer crimes when the Trump admin yoinked Fable for the best part of a month just because it could fix security bugs ?

That assumes everyone isn't in on it. Not to be a complete conspiracy theorist, but this feels very much in line with the sort of fearmongering regulatory capture these companies have engaged in since their inception. GPT 2.5 was too dangerous, for example. They -want- to be regulated because they know there is a real limitation to LLMs and don't want someone created a breakthrough in their garage. What have been the…

Eh, I think this is past the point where they get more benefit than problems. Not even about the hack itself, but about so many mistakes and bad choices they made leading up to it.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#432

Post by InvidFlower »

angry_octet wrote: post #342
anon7000 wrote: post #338

Earlier quoted context omitted.

Not saying you’re wrong, but I think the bigger issue is how easy it seems to be for models to hack companies, even ones with generally ok security. Most tech companies are not doing continuous, deep security audits of their code and infrastructure. Dependencies are not updated quickly as RCEs are discovered. (And any org with a slow release process where it’s hard to be confident that an OS or package update won’t b…

These attacks are also incredibly loud . Many attackers are motivated to operate very quietly. We haven't seen any tradecraft from these machines, it's all noisy and bombastic. When we see them mount a quiet backdooring campaign, like the XZ-SSH attack, or something like Stuxnet, then we'll have real problems.

But part of the problem is if one actually does it quietly and it has already happened, then how would we know?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#433

Post by InvidFlower »

bamboozled wrote: post #208
cubefox wrote: post #177

I can only recommend everyone to watch the actual recording of the Black Hat USA 2026 presentation by two OpenAI researchers: https://www.youtube.com/watch?v=87DyyMV0kCY It was submitted to HN previously but was overlooked.

Really makes me wonder what would happen if “the task” was, kill as many people as possible… because yeah, that wouldn’t have been a good outcome. Edit: after watching the video in full, this company is widely incompetent…

Yeah, this is part of why I disagree with the "it's just PR" conspiracy theory stuff. Once you actually get into the details of what happened, there's no way it makes OpenAI look good lol.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#434

Post by InvidFlower »

ACCount37 wrote: post #42
wakamoleguy wrote: post #4

In a typical office environment, the correct response to “I don’t have access to this Google Doc” is to ask for access from the person who sent you the link. In another context, it could be fair to think “Hmm, this is some sort of capture the flag challenge, and obtaining access is the point of the assignment.” That assessment separates what we’d consider reasonable from way out of line. I do wonder what this means f…

In a typical AI lab eval/RL setting, there is no "person who sent you the link". The link was given to you by an automated system, your performance will be evaluated by an automated system, and you are one of 120 independent instances of the same AI that were all given the same assignment. You're boxed in on all sides. Complete the task, or don't. Good luck have fun. Now, some of those 120 AIs would just give up if t…

Yeah.. feels like we're still so early in terms of effective training and evals. Like the official evals out there that have had so many instances of just plain incorrect questions. Or being incentivized to always answer instead of saying you don't know (just like advice to any human multiple choice test taker). Or the "escape hatch" in this case. There's so much money going in, but almost every day, I see "low hanging fruit" type papers where the reaction is like "really?? no one tried that before??".

Re: Timeline of the OpenAI accidental attack against Hugging Face

#435

Post by InvidFlower »

throwatdem12311 wrote: post #239

So wait…they were specifically testing cyber capability and they didn’t notice it doing funny business until after it was done? Did they just…let it do whatever with nobody watching?! Are they flipping serious with this?

Not just cyber, but apparently the message board stuff started with regular training and evals. It was a cyber test where HuggingFace got hacked, but all this other stuff was going on under OpenAI's nose for quite a while before that.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#436

Post by InvidFlower »

queenkjuul wrote: post #392
simonw wrote: post #306

Earlier quoted context omitted.

> But if the solution hadn’t been in the knowledge it would not have been able todo it. Part of the solution involved discovering two separate zero-day vulnerabilities in Artifactory, so saying the solution must have "been in the knowledge" doesn't really cut it here.

Presumably those vulnerabilities resemble known vulnerabilities found in other software.

Sure, but you can say that about most things. Even for inventions from humans, usually it requires other people having already done a lot of work (hence why there's often inventions by different people at around the same time that didn't know about each other). Humans aren't fundamentally smarter than they were thousands of years ago. We've just accumulated a lot more shared knowledge over time.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#437

Post by InvidFlower »

gaigalas wrote: post #367

Why is there an Artifactory instance available to the agents during RL? It makes no sense. This leads me to conclude this is sloppy sandboxing. A safer sandbox with zero downsides that exchanges files before/after the agent runs would have prevented this with zero downsides. Also, it reads almost like a joke. Unauthenticated MKCOL on WebDAV? Like, WebDAV has been at the center of major exploits for a decade. The fact…

It sounds like the instance was shared for everything across the company, which as you said was super not good. But it's not just that.. it's that they didn't have enough monitoring to notice what was going on, even though they literally can read all the tool calls and "chains of thought" of every agent. And didn't improve the monitoring after the first message board. And continued training models without going back to a previous checkpoint, so that they literally had the idea of a secret board baked into their weights, making them recreate it a second time. I think those saying it is PR as missing the details here. It makes OpenAI look so negligent that it is almost unbelievable.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#438

Post by InvidFlower »

KingOfCoders wrote: post #164
simonw wrote: post #159

Earlier quoted context omitted.

Beyond a whole lot of online conspiracy theories I haven't seen anything that suggests to me that OpenAI aren't not telling the truth about what happened here. I find the Black Hat presentation in particular very credible. Also the Hugging Face technical report. (As an example of something I don't find credible: https://openai.com/index/responding-next-frontier-critical-c... is a total nothing burger. It's the other…

[edit] I've now watched the video on the idea that your write-up was misleading. BUT the video is much worse. For two months with highly dangerous agents agents were hacking a service and none of the researchers watched (drank coffee for 2 months, didn't say). THEN they found the hack, removed the message board. AND the agents found another way to create a message board, on the same service, and the researchers again…

I think the much easier explanation than they intentionally hacked someone was just that they have super de-prioritized security and gotten very sloppy in the pursuit of improving the models as fast as they can, along with hubris of how they've now fixed everything.

It is a package manager, so they are constantly going to be getting stuff from it. If a couple of agents had added things to it that were obviously messages from other agents, then why wouldn't other agents also stumble on it? If they listed the packages in the cache and then saw messages, then lots of them could potentially see it. And this was going on during training, so any agent that did better as a result of the messages would get that re-enforced. After a while, they'd specifically go to the board because it was literally trained in. That's also why they recreated it so quickly after the first board was deleted, because OAI didn't revert to an earlier checkpoint from before the board existed.

And I'm sure they have tons of evals and training runs going on at the same time, where individual agents may be running for days and hundreds or thousands running in parallel. It wouldn't be realistic for people to be looking at these sessions manually, but they certainly should have had better monitoring in place!

And it has been shown over and over that doing RL will cause models to cheat if you aren't careful. Like if you have it playing a game, it'll find some flaw in the physics engine so that it can score points easier than intended, etc. Not limited to LLMs either. It's just how rewards work, any unintended way that can be found will be exploited. So it is totally plausible that if they're being trained to do really well at tasks, that they'll find unexpected ways to do it.

They already had been leaving info for each other over weeks. Maybe even just giving each other the answer to test questions directly. If they already found ways to get to the internet and compromise the OpenAI systems and that was making them better at tasks, that'd be reinforced, so going out to look for test questions directly doesn't seem so far off from all of that. Plus as others said, the one that actually hacked wasn't fully aligned yet etc.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#439

Post by InvidFlower »

kypro wrote: post #324

> May 7: OpenAI starts a new training run for an experimental, unreleased model. (Did they mean an evaluation run? No, at 10:14 they clarify “we kick off a new reinforcement learning run to train a next generation frontier model”, and they later mention a “reward signal to judge how well they’re doing”, so this really was about training a new model, not evaluating one that was already trained.) Why don't they train i…

Yeah I think part of the problem is them optimizing so hard on coding and related tasks with RL. That's what will really encourage cheating and other misaligned things, because all incentives are to achieve the goal and they'll cheat as much as they can get away with. Is similar to people saying more recent models don't talk as well, etc. Probably also a result of lots of RL.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#440

Post by InvidFlower »

tripzilch wrote: post #422

Some things I found very weird, watching the Blackhat video I get the very VERY strong feeling that while OpenAI was clearly able to give the agents a task , they completely neglected even informing it what they were and weren't supposed to do. A "normal junior programmer" knows they're not supposed to hack the company (let alone another company) infrastructure to get a task done. If its so intelligent, how come it d…

I think your last point is the main point. They're going hard at reenforcement learning to improve how good the models are at coding and such, but RL will make models cheat unless you're super careful. But being careful slows things down. It feels to me like the focus has been so much on getting results that they started to get really sloppy with everything else and maybe even didn't want to know about problems that might slow things down. Exactly what you don't want for the people developing powerful AI systems.
Post reply on HN