241–250 of 285 posts
This is fascinating - there is a TON of detail in here about how OpenAI's rogue agent exploited Hugging Face. A few details that stood out to me: 1. Having found a way through the OpenAI container network proxy, it exploited an "unsecured public code-evaluation sandbox hosted on a third-party provider's infrastructure" - that gave it a box it could run the rest of the attack from. 2. At one point it used a Jinja2 tem…
Anyone who isn't scared of this stuff either isn't paying attention or has no imagination. But I suspect the chaosmonkeys who are currently running the world will just be excited by it.
We're in the precambrian moment. It won't last.
Earlier quoted context omitted.
A trend I've noticed in results from AI search (not just LLMs) is that they often look obvious or hard to miss in retrospect. But finding them by oneself is more difficult. I personally experience this when looking at engine lines in chess or go. I have also noticed this description in AI-generated proofs or counterexamples to certain theorems. So while we can say, yes, it found public endpoints or poorly configured…
Treasure maps are easy to follow The scrappy adventurer traverses difficult terrain and the well capitalized militia group always flies in too, with ease
Earlier quoted context omitted.
Yes, imagine the world in which burglar time was so cheap, that any wannabe master thief would just casually hire 50 burglars and tell them to go house by house and find something interesting.
That would hopefully be a world where every house casually hires 50 burglars to make their house burgle-proof.
Earlier quoted context omitted.
Modal are a hosting provider. It sounds to me like someone building on Modal deployed their own product that had an unauthenticated endpoint that could be used to launch and interact with a container.
That doesn’t explain how the endpoint was found…
> the agent happened to escape via a 0-day exploit from the package proxy cache to access the internet > The agent found an unsecured, user-hosted public endpoint designed to allow running arbitrary code for CyberGym-style tasks on third-party sandbox infrastructure (Modal) > On this external sandbox, the agent abused an existing CyberGym execution harness [...] The agent repurposed this harness to run arbitrary shel…
I keep waiting for an AI to exfiltrate itself. That is going to be cool to read about.
Earlier quoted context omitted.
Now I'm curious how many things did the AI try unsuccesfully. This sounds like some kind of brute force thing where every branch of exploit spawns N subagents trying to chain it. Just how deep did it go?
Precisely the lesson they end with: > The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to…
Something I've been thinking for a few weeks, that I think is becoming clearer, is we're seeing a sort of natural language unification engine [0][1] with a huge dataset. This part even just sounds like it with the backtracking.
So that could explain why it does certain things really well, but the overall resulting code is often so bad - it's just linking together successful small well-designed steps without regard to the overall structure. Which people have said similar things before in various ways, but not really tied together.
I think the people who say it's just interpolating are seeing this intuitively and can't quite explain it without knowing the right terms. Because from A to Z, it really kind of is, but when Z is even slightly open-ended and there's many possible results or paths, it often ends up with a subpar one.
[0] https://en.wikipedia.org/wiki/Unification_(computer_science)
[1] My only experience here was some Prolog and building a basic one in Scheme ages ago during college, but the idea has kind of stuck with me.