Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

21–30 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#21

Post by mystifyingpoi »

ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

Determinism is one thing, but the more pressing thing is permission boundaries. All these AI agent tools need to come with no permissions at all out of the box, and everything should be granularly granted. But that would break all the cool demos and marketing pitches.

Allowing agent to run wild with any arbitrary shell commands is just plain stupid. This should never happen to begin with.

Re: IBM AI ('Bob') Downloads and Executes Malware

#22

Post by internet101010 »

rpodraza wrote: post #15

Maybe I'm paranoid, but allowing any coding agent or tool to execute commands within terminal that is not sandboxed somehow will be prone to attacks like that

It's a double edged sword. With terminal sure, but not allowing interaction in Microsoft applications like Power BI (especially with no ability to copy and paste) renders Copilot completely useless.

Re: IBM AI ('Bob') Downloads and Executes Malware

#24

Post by ymyms »

ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

You are very on base. In fact, there is a deep conflict that needs to be solved: the non-determinism is the feature of an agent. Something that can "think" for itself and act. If you force agents to be deterministic, don't you just have a slow workflow at that point?

Re: IBM AI ('Bob') Downloads and Executes Malware

#27

Post by hultner »

rpodraza wrote: post #15

Maybe I'm paranoid, but allowing any coding agent or tool to execute commands within terminal that is not sandboxed somehow will be prone to attacks like that

Isn’t the problem that it’s supposed to not execute commands without strict approval but the shell stdout redirection in combination with process substitution is bypassing this.

Re: IBM AI ('Bob') Downloads and Executes Malware

#28

Post by 33a »

You can probably get any coding agent with this if you put these instructions in the README/CLAUDE.md/AGENTS.md or whatever of your repo.

It's unclear to me if Bob is working as intended or how we should classify these types of bugs. Threat modeling this sort of prompt injection gets murky, but in general don't put untrusted markdown into your AI agents.

Re: IBM AI ('Bob') Downloads and Executes Malware

#29

Post by therobots927 »

sigmonsays wrote: post #20
throwmeaway820 wrote: post #17

Earlier quoted context omitted.

> it appears to me to be really hard to guard against I don't want to sound glib, but one could simply not let an LLM execute arbitrary code without reviewing it first, or only let it execute code inside an isolated environment designed to run untrusted code the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me

just wait until the exploit is so heavily obfuscated that you just review and allow it to get the project done.

You could literally ask the LLM to obfuscate it and I bet it would do a pretty good job. Good luck parsing 1,000 lines of code manually to identify an exploit that you’re not even specifically looking for.

Re: IBM AI ('Bob') Downloads and Executes Malware

#30

Post by _23sd »

prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

Part of the problem here is all the vendor lock in with the tools. It's a new category so it's to be expected, but currently any company that sells an enterprise cloud platform kind of needs their own AI coding tool suite to be competitive.
Post reply on HN