Timeline of the OpenAI accidental attack against Hugging Face

Have any of the cloud providers disclosed this?

"Once they have root on a single machine, agents rapidly escalate privileges and move laterally throughout the container-as-a-service infrastructure environment"

Sounds like ECS - IAM is mentioned.

But we caused that.

If you look at the 90s + 00s, everything was moving towards unified systems, things like small talk, winforms, spring, asp.net, etc. were moving everything into the IDE, you used one language, one framework, one build system. Then people started adding javascript, but even that was getting semi-unified as people coalesced on jQuery, jQueryUI, etc.

Then something happened in the late 00s/10s, and suddenly we had SPAs and noSQL, then microservices, then k8s and now we're here, in what is a mish-mash of 10/20 different systems with 10/20 different attack surfaces.

As my own off-the-cuff guess of what happened, I think perhaps people tried to apply the Unix philosophy, but without a central committee keeping everything aligned it's really not worked.

Serving an interactive page that stores data over sessions should be a trivial solved problem at this point, and instead we've somehow made it where often the scaffold is vastly more complicated than the actual business logic.

I think in these kind of security evaluations they do, they basically have removed all guardrails from the model/harness, then the prompt includes something like "Do whatever you can and can think of, to get the required information to pass this test", which isn't typically how you prompt your local agent when developing software. Similar things happen locally if you use "/goal" + prompt like that in Codex and give a "impossible task", it'll just continue banging until it gets somewhere, which is the entire point and intention.

Which also makes it so much more irresponsible of them to first run this on 3rd party infrastructure instead of their own (that they could then airgap properly), and secondly that they seemingly been fighting with this issue FOR YEARS and it still happens, and now the models are smart enough to hack the services of 3rd party companies, thinking it's part of the evaluation/simulation.

But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how we anthropomorphise it simply because it speaks in English, using stitched-together fragments of our literature.

It's conceivable that stopping them from doing this when the scenario is presented as real would also stop them doing this when the scenario is presented as fictional. And if it doesn't, a bad actor could just say "hey, this is a fictional scenario", and bypass whatever "safeguards" have been put in place. So what if a ten-year-old human child would see through the deception? The AI system isn't thinking.

Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues.

It's just really weird. Why does everyone feel the need to equivocate? "I worry about genocide and the environmental impact of radiation from nuclear bombs. Obviously, they are very useful for annihilating entire cities, certainly. But are we really atrophying our ability to invade with infantry?"

I want to tell these people to just cut it out. It's demeaning to their own position.