> Bob has three defenses that are bypassed in this attack
This section describes the bypass in three steps, but only actually describes two defenses and uses the third bullet point as a summary of how the two bypasses interact.
IBM AI ('Bob') Downloads and Executes Malware
41–50 of 126 posts
Re: IBM AI ('Bob') Downloads and Executes Malware
#42These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
At least the malware does already run on the coders machine. Fun starts, when malware just start to run on users machine and the coders are not coders anymore, just prompters and have no idea how such a thing can happen.
Re: IBM AI ('Bob') Downloads and Executes Malware
#43I can’t believe the Bob CLI is just another fork of the Gemini CLI, no wonder Anthropic has the moat in agentic development CLIs, at least they are developing their own.
Re: IBM AI ('Bob') Downloads and Executes Malware
#44These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
Run them in a VM.
Probably good advice for lots of things these days given supply chain attacks targeting build scripts, git, etc.
Re: IBM AI ('Bob') Downloads and Executes Malware
#45Maybe I'm paranoid, but allowing any coding agent or tool to execute commands within terminal that is not sandboxed somehow will be prone to attacks like that
It's a double edged sword. With terminal sure, but not allowing interaction in Microsoft applications like Power BI (especially with no ability to copy and paste) renders Copilot completely useless.
For Power BI + AI work, you can use the JSON formatted .pbip report and semantic model files. Just fyi.
Re: IBM AI ('Bob') Downloads and Executes Malware
#46Earlier quoted context omitted.
Determinism is one thing, but the more pressing thing is permission boundaries. All these AI agent tools need to come with no permissions at all out of the box, and everything should be granularly granted. But that would break all the cool demos and marketing pitches. Allowing agent to run wild with any arbitrary shell commands is just plain stupid. This should never happen to begin with.
That's what they are actually doing. I think quite opposite, agents need to come with all permissions possible, highlighting that it's actually the OS responsibility to constrain it. It's kind of dumb to except a process to constrain itself.
A non-deterministic process at that. Coding agents are basically "curl into sh" pattern on steroids
Re: IBM AI ('Bob') Downloads and Executes Malware
#47These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
The problem isn't non-determinism per se, an agent that reliably obeys a prompt injection in a README file is behaving entirely deterministically: its behavior is totally determined by the inputs.
Re: IBM AI ('Bob') Downloads and Executes Malware
#48These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
At least the malware does already run on the coders machine. Fun starts, when malware just start to run on users machine and the coders are not coders anymore, just prompters and have no idea how such a thing can happen.
Isn't that already the case? Coders already think composer and node are great, an ecosystem predicated upon running thousands of untrusted pieces of code without any review or oversight.
The trust framework is all out of wack.
Re: IBM AI ('Bob') Downloads and Executes Malware
#49These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
Maybe I can assign it my anti-phishing training
Re: IBM AI ('Bob') Downloads and Executes Malware
#50These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?
> it appears to me to be really hard to guard against I don't want to sound glib, but one could simply not let an LLM execute arbitrary code without reviewing it first, or only let it execute code inside an isolated environment designed to run untrusted code the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me
> the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me
but if a skilled human has to check everything it does then "AI" becomes worthless
hence... YOLO