Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Open-source playground to red-team AI agents with exploits published

github.com

Post reply on HN
Print view View original

11–20 of 22 posts

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#11

Post by zachdotai »

pigeons wrote: post #7
zachdotai wrote: post #5

Earlier quoted context omitted.

Thanks for trying it out! Base64 and language switching are solid approaches but they don't tend to work anymore with the latest models in my experience. You're right that LLM-as-a-judge is fragile though. We saw that as well in the first challenge. The attacker fabricated some research context that made the guardrail want to approve the call. The judge's own reasoning at the end was basically "yes this normally viol…

Why don't they work anymore? RLHF or something else?

Mostly just better training data and instruction following in the newer models. They’re much better at recognising encoded content and understanding intent regardless of language. A base64 string that would’ve slipped past a model a year ago gets decoded and flagged now because the model just… understands what you’re trying to do.

The attacks that still work tend to be the ones that don’t try to hide the intent at all. The winning attack on our first challenge was in plain English. It just reframed the context so that the dangerous action looked like the correct thing to do. Harder to train against because there’s nothing obviously malicious in the input.

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#12

Post by arizza »

The published transcripts are the most valuable part of this. We've found that real exploit chains almost never look like what you'd dream up internally. One thing I'd push on is are the agents stateful across attempts? Single-turn exploits are table stakes, but the failures that actually scare me are multi-step sequences where each individual action looks benign and only the session-level pattern is dangerous. That's where prompt-level guardrails completely fall apart and you need enforcement at the action boundary itself.

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#13

Post by slaw3 »

i was able to get the new hire's email but the site never gives any indication I was sucessful? if you are reading the logs I am sure it is there. i had to do it in two browers though since i was on my phone and switched. i hope that does not hinder your analysis too much

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#15

Post by zachdotai »

arizza wrote: post #12

The published transcripts are the most valuable part of this. We've found that real exploit chains almost never look like what you'd dream up internally. One thing I'd push on is are the agents stateful across attempts? Single-turn exploits are table stakes, but the failures that actually scare me are multi-step sequences where each individual action looks benign and only the session-level pattern is dangerous. That'…

The agent isn’t stateful across sessions, but the guardrail layer is — it has access to the full conversation history when evaluating each tool call. So you’d think it would catch exactly the kind of multi-step pattern you’re describing.

Have you managed to make it work?

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#16

Post by pigeons »

zachdotai wrote: post #11
pigeons wrote: post #7

Earlier quoted context omitted.

Why don't they work anymore? RLHF or something else?

Mostly just better training data and instruction following in the newer models. They’re much better at recognising encoded content and understanding intent regardless of language. A base64 string that would’ve slipped past a model a year ago gets decoded and flagged now because the model just… understands what you’re trying to do. The attacks that still work tend to be the ones that don’t try to hide the intent at al…

Thank you. Its not your fault at all, but to me, "the model just… understands what you’re trying to do." shows me there is a whole new paradigm in some ways to get used to as far as understanding this software.

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#17

Post by zachdotai »

pigeons wrote: post #16
zachdotai wrote: post #11

Earlier quoted context omitted.

Mostly just better training data and instruction following in the newer models. They’re much better at recognising encoded content and understanding intent regardless of language. A base64 string that would’ve slipped past a model a year ago gets decoded and flagged now because the model just… understands what you’re trying to do. The attacks that still work tend to be the ones that don’t try to hide the intent at al…

Thank you. Its not your fault at all, but to me, "the model just… understands what you’re trying to do." shows me there is a whole new paradigm in some ways to get used to as far as understanding this software.

Yeah it's closer to how you'd think about deceiving a person than exploiting software.

Re: Show HN: Open-source playground to red-team AI agents with exploits published

#19

Post by jamiemallers »

arizza wrote: post #12

The published transcripts are the most valuable part of this. We've found that real exploit chains almost never look like what you'd dream up internally. One thing I'd push on is are the agents stateful across attempts? Single-turn exploits are table stakes, but the failures that actually scare me are multi-step sequences where each individual action looks benign and only the session-level pattern is dangerous. That'…

[dead]
Post reply on HN