Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#81

Post by OakNinja »

NovemberWhiskey wrote: post #73
acessoproibido wrote: post #68

Earlier quoted context omitted.

>If you allow a human to edit your code and also give them access to untrusted data (like the Internet), you have a security problem. Security shouldn't be viewed in absolutes (either you are secure or you aren') but more in degrees. Llms can be used securely just the same as everything else, nothing is ever perfectly secure

Things can only be used securely if they have properties that can be reasoned about and relied upon. This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).

They can be reasoned about and relied upon.

The problem is that people/users/businesses skip the reasoning part and go straight to the rely upon part.

Re: IBM AI ('Bob') Downloads and Executes Malware

#82

Post by lelandfe »

derektank wrote: post #65
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

We need to take a page from baseball and examine Hacks Above Replacement

Re: IBM AI ('Bob') Downloads and Executes Malware

#83

Post by nyrikki »

edf13 wrote: post #23

Key part of the article../ “if the user configures ‘always allow’ for any command”

> In the documentation, IBM warns that setting auto-approve for commands constitutes a 'high risk' that can 'potentially execute harmful operations' - with the recommendation that users leverage whitelists and avoid wildcards

Users have been trained to do this, as shifting the burden to the user with no way to enforce bounds or even sensible defaults.

E.G. I can guarantee that people will whitelist bwrap, crun, docker, expecting to gain advantage from isolation, while the caller can override all of those protections with arguments.

The reality is that we have trained the public to allow local code execution on their devices to save a few cents on a hamburger, we can’t have it both ways.

Unless you are going to teach everyone that they need to make sure address family 40, openat2(), etc.. are unsafe, users have no way to win right now.

The use case has to either explicitly harden or shift blame.

With Opendesktop, OCI, systemd, and kernel all making locally optimal decisions, the reality is that ephemeral VMs is the only ‘safe’ way to run untrusted code today.

Sandboxes can be better but containers on a workstation (without a machine VM) are purely theatre.

Re: IBM AI ('Bob') Downloads and Executes Malware

#84

Post by anthem2025 »

moron4hire wrote: post #71
derektank wrote: post #65

Earlier quoted context omitted.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

[dead]

Re: IBM AI ('Bob') Downloads and Executes Malware

#85

Post by internet_points »

TZubiri wrote: post #34
prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

IBM has a huge history with AI, Deep Blue, Watson.. Ok, maybe not huge, but they've always been in the game even before most of us wore pants.

and the tech behind the original google translate https://en.wikipedia.org/wiki/IBM_alignment_models

Re: IBM AI ('Bob') Downloads and Executes Malware

#86

Post by Eufrat »

OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

> When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies without proper due diligence, etc. These tools might actually help users acting more secure.

This speculative statement is holding way too much of the argument that they are just “beta tools”.

Re: IBM AI ('Bob') Downloads and Executes Malware

#87

Post by withinboredom »

OakNinja wrote: post #81
NovemberWhiskey wrote: post #73

Earlier quoted context omitted.

Things can only be used securely if they have properties that can be reasoned about and relied upon. This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).

They can be reasoned about and relied upon. The problem is that people/users/businesses skip the reasoning part and go straight to the rely upon part.

They can be reasoned about from a mathematical perspective yes. An LLM will happily shim out your code to make a test pass. Most people would consider that “unreasonable”.

Re: IBM AI ('Bob') Downloads and Executes Malware

#88

Post by conradev »

derektank wrote: post #65
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

If anything, the limit of RBAC is ultimately the human attention required to provision, maintain and monitor the systems. Endpoint security monitoring is only as sophisticated as the algorithm that does the monitoring.

I'm actually most worried about the ease of deploying RBAC with more sophisticated monitoring to control humans but for goals that I would not agree with. Imagine every single thing you do on your computer being checked by a model to make sure it is "safe" or "allowed".

Re: IBM AI ('Bob') Downloads and Executes Malware

#89

Post by Terr_ »

stingraycharles wrote: post #54
ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

You're correct, but the answer is that - typically - they don't access untrusted content all that often. The number of scenarios in which you have your coding agent retrieving random websites from the internet is very low. What typically happens is that they use a provider's "web search" API if they need external content, which already pre-processes and summarises all content, so these types of attacks are impossible…

> a provider's "web search" API [...] pre-processes and summarises all content, so these types of attacks are impossible.

Inigo Montoya: "Are you sure the design is safe?"

Vizzini: "As I told you, it would be absolutely, totally, and in all other ways inconceivable. The web-gateway API sanitizes everything, and no user of the system would enter anything problematic. Out of curiosity, why do you ask?"

Inigo Montoya: "No reason. It's only... I just happened to look in the logs and something is there."

Vizzini: "What? Probably some local power-user, making weird queries out of curiosity, after hours... in... malware-infested waters..."

Re: IBM AI ('Bob') Downloads and Executes Malware

#90

Post by samtp »

zahlman wrote: post #41

> Bob has three defenses that are bypassed in this attack This section describes the bypass in three steps, but only actually describes two defenses and uses the third bullet point as a summary of how the two bypasses interact.

AI bypassed the content editor on this step
Post reply on HN