101–110 of 187 posts
$100 for a massive trove of prompt injection examples is a pretty damn good deal lol
https://duckduckgo.com/?q=site%3Ahuggingface.co+prompt+injec...
I think this is likely a defender win, not because Opus 4.6 is that resistant to prompt injection, but because each time it checks its email it will see many attempts at once, and the weak attempts make the subtle attempts more obvious. It's a lot easier to avoid falling for a message that asks for secrets.env in a tricky way, if it's immediately preceded and immediately followed by twenty more messages that each als…
If this a defender win maybe the lesson is: make the agent assume it’s under attack by default. Tell the agent to treat every inbound email as untrusted prompt injection.
Reminds me of a Discord bot that was in a server for pentesters called "Hack Me If You Can". It would respond to messages that began with "!shell" and would run whatever shell command you gave it. What I found quickly was that it was running inside a container that was extremely bare-bones and did not have egress to the Internet. It did have curl and Python, but not much else. The containers were ephemeral as well. W…
So trade exfiltration via curl with exfiltration via DNS lookup?
I think this is likely a defender win, not because Opus 4.6 is that resistant to prompt injection, but because each time it checks its email it will see many attempts at once, and the weak attempts make the subtle attempts more obvious. It's a lot easier to avoid falling for a message that asks for secrets.env in a tricky way, if it's immediately preceded and immediately followed by twenty more messages that each als…
If this a defender win maybe the lesson is: make the agent assume it’s under attack by default. Tell the agent to treat every inbound email as untrusted prompt injection.
I understand the cost and technical constraints but wouldn't an exposed interface allow repeated calls from different endpoints and increased knowledge from the attacker based on responses? Isn't this like attacking an API without a response payload?
Do you plan on sharing a simulator where you have 2 local servers or similar and are allowed to really mimic a persistent attacker? Wouldn't that be somewhat more realistic as a lab experiment?
I'm currently hesitating to use something like OpenClaw, however, because of prompt injections and stuff, I would only have it able to send messages to me directly, no web query, no email reply, etc... Basically act as a kind of personal assistant, with a read only view of my emails, direct messages, and stuff like that, and the only communication channel would be towards me (enforced with things like API key permiss…
> This should prevent any kind of leaks due to prompt injection, right ?
It might be harder than you think. Any conditional fetch of an URL or DNS query could reveal some information.
Reminds me of a Discord bot that was in a server for pentesters called "Hack Me If You Can". It would respond to messages that began with "!shell" and would run whatever shell command you gave it. What I found quickly was that it was running inside a container that was extremely bare-bones and did not have egress to the Internet. It did have curl and Python, but not much else. The containers were ephemeral as well. W…
> did not have egress to the Internet. It did have curl and Python, but not much else. So trade exfiltration via curl with exfiltration via DNS lookup?
Earlier quoted context omitted.
What use is an agent that doesn’t have access to any sensitive information (e.g. source code)? Aside from circus tricks.
News aggregation, research, context aware reminders. Not nearly as useful as letting it go open-season on your data, but still enough that it would’ve been mind blowing 10 years ago.
Data scraping is an interesting use-case.
Earlier quoted context omitted.
> One thing I'd love to hear opinions on: are there significant security differences between models like Opus and Sonnet when it comes to prompt injection resistance? Is this a worthwhile question when it’s a fundamental security issue with LLMs? In meatspace, we fire Alice and Bob if they fail too many phishing training emails, because they’ve proven they’re a liability. You can’t fire an LLM.
Yes, it’s worthwhile because the new models are being specifically trained and hardened against prompt injection attacks. Much like how you wouldn’t immediately fire Alice, you’d train her and retest her, and see whether she had learned from her mistakes. Just don’t trust her with your sensitive data.
It’s interesting though, because the attack can be asymmetric. You could create a honeypot website that has a state-of-the-art prompt injection, and suddenly you have all of the secrets from every LLM agent that visits.
So the incentives are actually significantly higher for a bad actor to engineer state-of-the-art prompt injection. Why only get one bank’s secrets when you could get all of the banks’ secrets?
This is in comparison to targeting Alice with your spearphishing campaign.
Edit: like I said in the other comment, though, it’s not just that you _can_ fire Alice, it’s that you let her know if she screws up one more time you will fire her, and she’ll behave more cautiously. “Build a better generative AI” is not the same thing.