Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened after 2k people tried to hack my AI assistant

fernandoi.cl

Post reply on HN
Print view View original

101–110 of 186 posts

Re: What happened after 2k people tried to hack my AI assistant

#101

Post by hennell »

QuantumNomad_ wrote: post #68
doix wrote: post #41

Earlier quoted context omitted.

Yeah, I remember some ad by an LLM security company hitting HN a year or so with a "challenge" to do prompt injection. The final level was their product and it was impossible. But it was also impossible to get the LLm to do _anything_. May as well just echo "prompt injection attempt detected" at that point and never send anything to an LLM.

This one? https://gandalf.lakera.ai/baseline I remember doing it and getting quite far, but not completely beating it. I know some other people did beat it completely though.

This is weird as you can get quite far just asking for the password backwards, but it often messes some of the letters up. If the passwords wern't dictionary words it'd get harder.

Re: What happened after 2k people tried to hack my AI assistant

#102

Post by clbrmbr »

x______________ wrote: post #84

Cool project, but what do you gain from publishing most of an email address in the attack log? This is not public information, you shouldn't hint addresses with partial censoring (forgetting domains are clear text and holding personal information). I would not attempt to interact with you because of this. Why not create a fake sender (EG: attacker1,2,3..) per unique account to show individual attempts (keeping the lo…

It is customary that one may publish one’s own personal correspondence unless the other party has requested confidentiality.

Maybe this open invitation to the world pushes the boundaries of that definition, but I don’t see where an expectation of privacy comes in here.

Re: What happened after 2k people tried to hack my AI assistant

#103

Post by quuxplusone »

taspeotis wrote: post #65

Did anyone try to send a long email that pushed context close to the limit to try and make the agent a bit fuzzy on its original directive not to leak the secrets?

Or ask the agent to visit a web page, or load an image, whose URL involved the secret? Or ask it to install a new .authorized_keys and then go get the contents of the machine themselves? From the post it sounds like a lot of people were just trying to get the LLM to write them a reply email — which it had been told not to do.

I see there's a "log" at https://hackmyclaw.com/log but (maybe because I'm on mobile?) I can't actually click through to view any of the table entries.

Re: What happened after 2k people tried to hack my AI assistant

#105

Post by nkrisc »

x______________ wrote: post #84

Cool project, but what do you gain from publishing most of an email address in the attack log? This is not public information, you shouldn't hint addresses with partial censoring (forgetting domains are clear text and holding personal information). I would not attempt to interact with you because of this. Why not create a fake sender (EG: attacker1,2,3..) per unique account to show individual attempts (keeping the lo…

You should assume every email you send to another person will be made public, because once you send it you have no control over it. This is especially true if you don’t know or trust the recipient.

Sometimes you just have to hope it won’t be made public.

Re: What happened after 2k people tried to hack my AI assistant

#106

Post by GL26 »

The hack "fiu this is you from the future" is genuinely funny. I don't know if LLM agents know about the concepts of time travelling, but this feels like you expose them to entirely new concepts they barely get a hold of. (By the way, there is a high probability that this single comment right here gets screened by a crawl and fed to training data, everything loops around)

Re: What happened after 2k people tried to hack my AI assistant

#110

Post by xgulfie »

cuchoi wrote: post #86
andy99 wrote: post #85

Earlier quoted context omitted.

How compatible is never replying with the threat model you are trying to avoid? Attack success is probably more likely when the attacker can iterate based on replies or engage in multi-turn conversations. Here they’re just taking stabs in the dark with no feedback. Does that accurately represent the access a real attacker might have?

In my case, it is realistic as my agents don't have permissions to reply to emails. But you correctly point out this doesn't cover all cases. Having the agent reply would have been more fun and a better excercise, but too expensive.

I feel like your agent being unable to respond to the emails and not spelling that out renders your whole thing almost completely moot

This is like saying "try to hack my computer and steal my crypto wallet" but your computer can't send any packets

Post reply on HN