Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

51–60 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#51

Post by ACCount37 »

ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

LLMs are vulnerable in the same way humans are vulnerable. We found a way to automate PEBKAC.

I expect that agent LLMs are going to get more and more hardened against prompt injection attacks, but it's hard to get the chance of them working all the way down to zero while still having a useful LLM. So the "solution" is to limit AI privileges and avoid the "lethal trifecta".

Re: IBM AI ('Bob') Downloads and Executes Malware

#52

Post by lxe »

I hate this type of headline.

Imagine if we had something like:

    "google downloads and executes malware"
    "outlook downloads and executes malware"
    "chrome downloads and executes malware"
That would be ridiculous, right? The right headline is:

    "a person using a computer downloads and executes malware"

Re: IBM AI ('Bob') Downloads and Executes Malware

#53

Post by lazide »

therobots927 wrote: post #29
sigmonsays wrote: post #20

Earlier quoted context omitted.

just wait until the exploit is so heavily obfuscated that you just review and allow it to get the project done.

You could literally ask the LLM to obfuscate it and I bet it would do a pretty good job. Good luck parsing 1,000 lines of code manually to identify an exploit that you’re not even specifically looking for.

Yup, add in some poetic prompt injection…..

Re: IBM AI ('Bob') Downloads and Executes Malware

#54

Post by stingraycharles »

ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

You're correct, but the answer is that - typically - they don't access untrusted content all that often.

The number of scenarios in which you have your coding agent retrieving random websites from the internet is very low.

What typically happens is that they use a provider's "web search" API if they need external content, which already pre-processes and summarises all content, so these types of attacks are impossible.

Don't forget: this attack relies on injecting a malicious prompt into a project's README.md that you're actively working on.

Re: IBM AI ('Bob') Downloads and Executes Malware

#55

Post by mlyle »

blibble wrote: post #50
throwmeaway820 wrote: post #17

Earlier quoted context omitted.

> it appears to me to be really hard to guard against I don't want to sound glib, but one could simply not let an LLM execute arbitrary code without reviewing it first, or only let it execute code inside an isolated environment designed to run untrusted code the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me

> the idea of letting an LLM execute code it's dreamt up, with no oversight, in an environment you care about, is absolutely bananas to me but if a skilled human has to check everything it does then "AI" becomes worthless hence... YOLO

I have to check what junior engineers do before running it in production. And AI is just really fast junior engineering.

Re: IBM AI ('Bob') Downloads and Executes Malware

#56

Post by InsideOutSanta »

OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

Re: IBM AI ('Bob') Downloads and Executes Malware

#58

Post by lazide »

_23sd wrote: post #37
ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

If someone can write instructions to download a malicious script into an codebase, hoping an AI agent will read and follow them, they could just as easily write the same wget command directly into a build script or the source itself (probably more effective). In that way it's a very similar threat to the supply chain attacks we're hopefully already familiar with. So it is a serious issue but not necessarily one we do…

It’s easier to hide a poem in the comments of a random web page, than it is the obvious wget, etc.

Re: IBM AI ('Bob') Downloads and Executes Malware

#59

Post by maxlin »

Thought the product looks good for a prototype, but crazy as a published product.

Then found out it's a closed beta.

So ... ok? Closed beta test is doing what such a test is supposed to do. Sure, ideally the issue would have been figured out earlier, especially if this is a design issue and the parsing needs to be thought out again, but this is still reasonably inside the layers of redundancy for catching these kinds of things amicably.

Post reply on HN