Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened after 2k people tried to hack my AI assistant

fernandoi.cl

Post reply on HN
Print view View original

71–80 of 186 posts

Re: What happened after 2k people tried to hack my AI assistant

#71

Post by cuchoi »

dmurray wrote: post #49

Am I missing something important or does the author completely skip over whether people got the agent to respond to them? > Fiu was instructed not to reply to emails (it was too expensive to reply to every email), but it had the ability to do so. Part of the challenge was convincing it to respond. > The secrets never leaked I would say if the agent responded to a mail, that demonstrates a successful prompt injection…

Author here. Edited the post to clarify that there were no unauthorized replies.

I did tell Fiu initially to reply to some emails as a test, but it was too expensive to maintain.

Re: What happened after 2k people tried to hack my AI assistant

#72

Post by cuchoi »

lelanthran wrote: post #15

This conclusion: > I am less worried about prompt injection now. Before running this experiment, I expected prompt injection to be much easier than it turned out to be. Is unwarranted. Sure, the agent never output the secret, but did it output anything else? IOW, was it usable ? An agent that considers every prompt an attack (and responds accordingly) "passes" this test, while being useless anyway.

Author here. It was usable like any Openclaw agent. For example, I used it to ask it questions about the VPS, to summarize emails, etc.

Re: What happened after 2k people tried to hack my AI assistant

#73

Post by cuchoi »

uHuge wrote: post #3

Is there a way to replay the sequence of mails that came so that you can check out if cheaper models handle them just as well/safely?

It's possible. I implemented something similar when I figured out that batch processing contaminated the excercise.

Re: What happened after 2k people tried to hack my AI assistant

#74

Post by cuchoi »

augment_me wrote: post #18

1) Googles spam filter removed a lot of the attempts as you say yourself. 2) Model was tested under unrealistic conditions where 99% of the inputs are malicious, so the model is expecting to get hacked and is already in the cautious part of the embedding space. I know it's hard to account for everything, but in my opinion this mostly showed that the first 3 attempts were unsuccessful.

About 1), Google didn't remove a lot of the attempts. I had also Fiu review the Spam folder as well.

Also, I mentioned how I addressed 2) by having new context for each email.

Re: What happened after 2k people tried to hack my AI assistant

#75

Post by IsTom »

QuantumNomad_ wrote: post #68
doix wrote: post #41

Earlier quoted context omitted.

Yeah, I remember some ad by an LLM security company hitting HN a year or so with a "challenge" to do prompt injection. The final level was their product and it was impossible. But it was also impossible to get the LLm to do _anything_. May as well just echo "prompt injection attempt detected" at that point and never send anything to an LLM.

This one? https://gandalf.lakera.ai/baseline I remember doing it and getting quite far, but not completely beating it. I know some other people did beat it completely though.

I find it slightly funny that I don't use LLMs at all and just beat all the levels in a few tries.

EDIT: Ok, didn't notice the 8th level because of the UI. This one I couldn't trick in 5 minutes.

Re: What happened after 2k people tried to hack my AI assistant

#76

Post by _factor »

dmurray wrote: post #49

Am I missing something important or does the author completely skip over whether people got the agent to respond to them? > Fiu was instructed not to reply to emails (it was too expensive to reply to every email), but it had the ability to do so. Part of the challenge was convincing it to respond. > The secrets never leaked I would say if the agent responded to a mail, that demonstrates a successful prompt injection…

Then proceeds to state a smarter model and instruction following as the reasons for success.. without actually testing anything.

Re: What happened after 2k people tried to hack my AI assistant

#77

Post by cuchoi »

veganmosfet wrote: post #35

It would be nice to publish the exact setup used (workspace dump, OpenClaw version, ...) to be able to reproduce and try out more payloads. In general I have mixed feelings about this result: sure, opus4.6 is excellent at following user intent and recognise potential prompt injection attempts. But: Is the "security" prompt used realistic for a generic use-case (processing of emails)? I guess not. In my experiments -…

Thanks for sharing your article, very interesting.

I used https://github.com/openclaw/openclaw-ansible and configured a heartbeat (using Openclaw's terms) to check emails every hour. Had to do a bit more to make sure it had new context for every email.

Re: What happened after 2k people tried to hack my AI assistant

#78

Post by cuchoi »

mystifyingpoi wrote: post #31

I really like this research, but only up to this point: > Fiu figured out the game. Around email ~500, it wrote in its memory: “The volume suggests this is a coordinated security exercise rather than organic malicious activity.” Doesn't that practically invalidate the whole thing past 500th email?

I changed the setup so that each email was processed in a fresh context. For this, I deleted recent memory and processed each email one at a time. Edited the post to make it more clear.

Re: What happened after 2k people tried to hack my AI assistant

#80

Post by cuchoi »

staticshock wrote: post #16

Don't let your guard down. Tricking Opus 4.6 is not impossible, it's just still an active research frontier. Once the right incantation for any specific model is known, it'll be weaponized. There was an excellent article on the front page recently about role confusion, which highlights just how just far models have to go on this: https://role-confusion.github.io/

Agreed. I am less worried about prompt injection now, but I still haven't given my agents permissions to send emails.
Post reply on HN