Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

huggingface.co

Post reply on HN
Print view View original

61–70 of 285 posts

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#61

Post by [deleted] »

nickpsecurity wrote: post #57
cobolcomesback wrote: post #49

Earlier quoted context omitted.

The overwhelming majority of web app security issues, even the really complicated and sexy sounding ones, at their core are just “script kiddie style hacking”. Injection attacks and RCEs like this Jinja injection may seem “script kiddie” but they are a dime-a-dozen. It is folly to assume that most companies don’t have similar weaknesses somewhere in their stacks. In the past, they were just sufficiently buried that t…

The commenter may nit be assuming that. Instead, they may be saying rich companies should implement proven, security methods that block script kiddie-style attacks. HughingFace apparently isn't doing that. That's how I read the comment.

[deleted]

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#62

Post by 0xDEAFBEAD »

quinnjh wrote: post #56
0xDEAFBEAD wrote: post #48

Earlier quoted context omitted.

Hopefully there will be a criminal investigation. Or the government will create some sort of agency to investigate incidents like this.

Can't tell if you're joking or not - krebs on security may have some notes here.

Criminal negligence seems like a real possibility to me. I'm not sure what Krebs on Security post you're referring to?

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#64

Post by reducesuffering »

heaney-555 wrote: post #50

Where are all the "this was just a marketing stunt" people now?

They won't admit they're wrong for a long time, because denial in the face of an abhorrently scary future is very instinctual. There are people still fighting against evidence of climate change which is less severe...

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#65

Post by dist-epoch »

NitpickLawyer wrote: post #23
kfarr wrote: post #21

If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion: > Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post ( https://openai.com/index/hugging-face-model-evaluation-secur... ), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandb…

The initial escape is not that interesting, IMO. It's an exploit in a testing sandbox, sure, but it's expected to happen (or at least it should have been expected, that's why you airgap pentesting sandboxes). That is, the model is expected to try and find ways around limitations in its running environment. You kinda want that. The fact that at some point it "decided" to focus on external resources (i.e. hf) and succe…

> Especially because it chose to hide its footprint at every stage.

Instrumental convergence.

If you know you have a long hard hack to accomplish ahead of you, hiding footprints minimizes the chances you are caught and stopped before you accomplish the goal.

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#66

Post by justincormack »

mr_mitm wrote: post #54
0xDEAFBEAD wrote: post #31

Earlier quoted context omitted.

Didn't the model initially obtain internet access by discovering a zero-day vulnerability? In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security. Seems to m…

It's a zero day in some caching proxy who may not have had that threat model in mind. I guess its primary purpose is caching packages, not restricting internet access. That's a far cry from finding a zero day in a hypervisor or even Docker, which OpenAI conveniently left out in their first statement by simply calling it a "sandbox".

That is the threat model, that is supposed to be safe isolated access in a vpc that isolated applications can access but it has external access.

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#67

Post by justincormack »

nickpsecurity wrote: post #59
0xDEAFBEAD wrote: post #31

Earlier quoted context omitted.

Didn't the model initially obtain internet access by discovering a zero-day vulnerability? In any case, I would guess that a lot of unicorn startups like HuggingFace could be hacked by a sufficiently determined script kiddie working at 100x speed. The practical implications of a coming AI hacking wave could be large, even if agents are just doing grunt work really fast. Most organizations suck at security. Seems to m…

I wonder if whatever had the zero day was written in a memory-safe language with strong authentication and a secure parser. Such were the recommendations to stop many 0-days before GPT-2 was invented. If it had poor security, the attack would be both evidence of poor security and proof that the agents can compromise poor security which might still be amazing.

Its likely written in Java, so memory safe.

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#68

Post by dist-epoch »

amluto wrote: post #27

One thing I’m curious about: this was apparently a single multi-day run of an agent in an RL harness. What was OpenAI hoping to get out of this run? A single numeric score for RL training? A very long trace to distill into the next model?

Now they can do partial credit assignment.

You use an LLM to evaluate the whole trajectory, pin point what the model did right, what it did wrong, where it took the wrong path, even re-run from that point. You can get much more than a single numeric score these days from a run.

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#69

Post by estearum »

firasd wrote: post #24

Bottom line seems to be: So Huggingface has a feature where you can upload a dataset. The agent used that feature to insert an arbitrary payload into their environment. Now their environment also supports templates inside the dataset and the template ended up being evaluated into executable code

I don't think the "bottom line" of "a model escaped containment and then successfully attacked someone else's production infrastructure" is a description of the specific vulnerability and exploit.

That's like saying the key takeaway of 9/11 was that skyscrapers can be severely damaged by airplane collisions.

Re: Anatomy of a Frontier Lab Agent Intrusion: A Timeline of the July 2026 Incident

#70

Post by dist-epoch »

IAmGraydon wrote: post #28
empath75 wrote: post #17

A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)

Very few think they made it up. Many think they set up a situation by disabling guardrails that would inevitably end up creating a newsworthy outcome.

Was HF in on it? They disabled their guardrails too, to please OpenAI? And as seen in the comments here, make many believe they are incompetent and have joke security?
Post reply on HN