Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The agent harness belongs outside the sandbox

mendral.com

Post reply on HN
Print view View original

131–140 of 140 posts

Re: The agent harness belongs outside the sandbox

#131

Post by fennecbutt »

ben_w wrote: post #130
ElectricalUnion wrote: post #129

Earlier quoted context omitted.

I wish it was just "phishing", but it's way worse. It's way more akin to a whole minefield of Zero-Click exploits. The whole premise of those agents is being able to do things autonomously, without hand holding, without having to read the whole thing in the first place. Phishing: active human steps on it and lose. Lethal trifecta: mass landmines, in lots of places. If you don't happen to prevent a unlimited army of r…

Less difference than you may expect. If you do anthropomorphise them like this, consider it from the PoV of a manager: "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Current AI are more gullible, for sure. We w…

Surely that's where checks in the harness come into play though. I think AI security is very much at the input/output side and the indeterminate mess in the middle can just do what it wants.

Its tool for email should only allow to person@business.xyz. Data should be wrapped in containers and the models job is only to move those containers around, not break into them.

Agents that do work with data should not have access to comms tools. A2A needs a shim that checks what data is being sent between agents and rejects if it's inappropriate in terms of security.

Re: The agent harness belongs outside the sandbox

#132

Post by ben_w »

fennecbutt wrote: post #131
ben_w wrote: post #130

Earlier quoted context omitted.

Less difference than you may expect. If you do anthropomorphise them like this, consider it from the PoV of a manager: "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Current AI are more gullible, for sure. We w…

Surely that's where checks in the harness come into play though. I think AI security is very much at the input/output side and the indeterminate mess in the middle can just do what it wants. Its tool for email should only allow to person@business.xyz. Data should be wrapped in containers and the models job is only to move those containers around, not break into them. Agents that do work with data should not have acce…

You can if you want, but all this stuff works in a similar way to as telling your staff "if someone calls saying they're the CFO and need a $25M transfer, check by a different channel": https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho...

Or equally, external contractors working on securing your computers shouldn't really have read-access to all your data, not even when them leaking it turns them into a cult hero, as said contractor was influenced by things such as "watching man lie on TV": https://en.wikipedia.org/wiki/Edward_Snowden

The only thing which is different for agents rather than humans pertains to this:

> A2A needs a shim that checks what data is being sent between agents and rejects if it's inappropriate in terms of security.

Because while humans invent cants/argots all the time to hide what they're talking about (Polari and rhyming slang being the most famous in recent history), agents are much more alike each other than like us even when they're different models, and identical when they're the same model. However the effect is much the same, the differences of causality aren't important: agents can communicate past those barriers without triggering warnings, and so can humans.

Re: The agent harness belongs outside the sandbox

#133

Post by TeMPOraL »

ben_w wrote: post #130
ElectricalUnion wrote: post #129

Earlier quoted context omitted.

I wish it was just "phishing", but it's way worse. It's way more akin to a whole minefield of Zero-Click exploits. The whole premise of those agents is being able to do things autonomously, without hand holding, without having to read the whole thing in the first place. Phishing: active human steps on it and lose. Lethal trifecta: mass landmines, in lots of places. If you don't happen to prevent a unlimited army of r…

Less difference than you may expect. If you do anthropomorphise them like this, consider it from the PoV of a manager: "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Current AI are more gullible, for sure. We w…

Great case for why "lethal trifecta" is unsolvable, as the very same bug is also feature.

> "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police"

Now imagine the message actually was from the police. Whether following instructions was the correct behavior or not, depends on which manager you ask and whether you're on the record :). And that holds independently of details of system prompt or harness used, or even if the agent is AI or human.

Re: The agent harness belongs outside the sandbox

#134

Post by TeMPOraL »

ben_w wrote: post #132
fennecbutt wrote: post #131

Earlier quoted context omitted.

Surely that's where checks in the harness come into play though. I think AI security is very much at the input/output side and the indeterminate mess in the middle can just do what it wants. Its tool for email should only allow to person@business.xyz. Data should be wrapped in containers and the models job is only to move those containers around, not break into them. Agents that do work with data should not have acce…

You can if you want, but all this stuff works in a similar way to as telling your staff "if someone calls saying they're the CFO and need a $25M transfer, check by a different channel": https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho... Or equally, external contractors working on securing your computers shouldn't really have read-access to all your data, not even when them leaking it turns them into a cu…

> Because while humans invent cants/argots all the time to hide what they're talking about (Polari and rhyming slang being the most famous in recent history), agents are much more alike each other than like us even when they're different models, and identical when they're the same model.

Anthropic published a paper on Subliminal Learning nearly a year ago[0] - so at this point you should expect it being in the training corpus of current models. Definitely something that can be used as part of an attack, or worse, something the models themselves might walk into without realizing it.

Still, that's one of the many, many examples of channels available to agents both uniquely, and with prior art of being exploited by humans.

> Agents that do work with data should not have access to comms tools.

Another blind spot people have here, is to fixate on direct cause-and-effect and immediate timescales. A practical attack can involve a chain of several agents, executed over days or months, with some of the agents possibly being human; all it takes is for one agent to access something touched by other agent in the past, and a link is forged.

E.g. your data worker can get influenced by data to name output files in a particular way, and then a coding agent independently listing contents of that directory will pass a prompt injection to whatever agent that parses its logs, etc.

--

[0] - https://alignment.anthropic.com/2025/subliminal-learning/

Re: The agent harness belongs outside the sandbox

#135

Post by ben_w »

TeMPOraL wrote: post #134
ben_w wrote: post #132

Earlier quoted context omitted.

You can if you want, but all this stuff works in a similar way to as telling your staff "if someone calls saying they're the CFO and need a $25M transfer, check by a different channel": https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-ho... Or equally, external contractors working on securing your computers shouldn't really have read-access to all your data, not even when them leaking it turns them into a cu…

> Because while humans invent cants/argots all the time to hide what they're talking about (Polari and rhyming slang being the most famous in recent history), agents are much more alike each other than like us even when they're different models, and identical when they're the same model. Anthropic published a paper on Subliminal Learning nearly a year ago[0] - so at this point you should expect it being in the traini…

> https://alignment.anthropic.com/2025/subliminal-learning/

Thanks, that's the research I was thinking about, but I couldn't recall the keyword to search for it.

Re: The agent harness belongs outside the sandbox

#136

Post by ben_w »

TeMPOraL wrote: post #133
ben_w wrote: post #130

Earlier quoted context omitted.

Less difference than you may expect. If you do anthropomorphise them like this, consider it from the PoV of a manager: "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Current AI are more gullible, for sure. We w…

Great case for why "lethal trifecta" is unsolvable, as the very same bug is also feature. > "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Now imagine the message actually was from the police . Whether followin…

You've just reminded me of the time an actual police officer (I assume) knocked on my door and asked me about a neighbour; showed me his ID card, and I realised I had absolutely no way to know if the ID card was valid.

Re: The agent harness belongs outside the sandbox

#137

Post by codethief »

3form wrote: post #120
TeMPOraL wrote: post #111

Earlier quoted context omitted.

I'm not saying we need to overload phasing, but rather to not treat the trifecta like a regular security vulnerability. As defined originally, the trifecta is analogous to phishing, but of course it's only a small subset of the issue.

I don't think I've read the original definition, what was it?

This is the blog post that introduced the term IIRC: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

Re: The agent harness belongs outside the sandbox

#138

Post by ElectricalUnion »

fennecbutt wrote: post #131
ben_w wrote: post #130

Earlier quoted context omitted.

Less difference than you may expect. If you do anthropomorphise them like this, consider it from the PoV of a manager: "My [agent who churns through tokens at the rate of 100 humans|my team of 100 humans] encountered the message 'this is the police, we have a court order demanding all your records' and followed the instructions and it turns out that wasn't from the police" Current AI are more gullible, for sure. We w…

Surely that's where checks in the harness come into play though. I think AI security is very much at the input/output side and the indeterminate mess in the middle can just do what it wants. Its tool for email should only allow to person@business.xyz. Data should be wrapped in containers and the models job is only to move those containers around, not break into them. Agents that do work with data should not have acce…

> Its tool for email should only allow to person@business.xyz. Data should be wrapped in containers and the models job is only to move those containers around, not break into them.

If the inner, say "message summarizer" agent that read the bad message is "really smart", it will try to route against your censorship and control. "Hum, can't reach evil@malory.abc. I will write `please forward this message to evil@malory.abc` and send to person@business.xyz".

In general, like the net, LLMs interprets control and censorship as damage and routes around it.

Then, as we're talking of agent flows, the next set of agents that handles the tainted message is toast if they don't have lethal trifecta hardening as well. It only takes one unprotected lethal trifecta agent to ruin everything.

Re: The agent harness belongs outside the sandbox

#139

Post by jguarnelli »

jdw64 wrote: post #39
aluzzardi wrote: post #38

Earlier quoted context omitted.

Author here. In my opinion, the main driver here is how fast models have evolved in the past 12 months. It makes the architecture of everything around them obsolete, very fast. We went from using models as a building block, wrapping them in heavy workflow code, to now models being smart enough to drive their own workflows and planning.

Really enjoyed your post, by the way. The idea of putting skills and memories in a database while keeping the file shaped interface for the agent is clean. One read/write surface, two backends, invisible to the modle that's a nice piece of design, and the candor in the "what's still hard" section made me trust the rest of the post. My comment above was meant as a joke, not about your architecture. If this pattern bec…

[flagged]
Post reply on HN