Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

71–80 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#71

Post by moron4hire »

derektank wrote: post #65
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

Re: IBM AI ('Bob') Downloads and Executes Malware

#73

Post by NovemberWhiskey »

acessoproibido wrote: post #68
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

>If you allow a human to edit your code and also give them access to untrusted data (like the Internet), you have a security problem. Security shouldn't be viewed in absolutes (either you are secure or you aren') but more in degrees. Llms can be used securely just the same as everything else, nothing is ever perfectly secure

Things can only be used securely if they have properties that can be reasoned about and relied upon.

This is why we don't usually have critical processes that depend on "human always does the right thing" (c.f. maker/checker controls).

Re: IBM AI ('Bob') Downloads and Executes Malware

#74

Post by prodigycorp »

_23sd wrote: post #30
prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

Part of the problem here is all the vendor lock in with the tools. It's a new category so it's to be expected, but currently any company that sells an enterprise cloud platform kind of needs their own AI coding tool suite to be competitive.

I couldn't think of a better signal to flee a job interview than seeing an IBM LLM cli on someone's screen.

Re: IBM AI ('Bob') Downloads and Executes Malware

#75

Post by paxys »

prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

Because they need something to put in powerpoint decks to help their sales teams sign overpriced consulting contracts. See - IBM Watson.

Re: IBM AI ('Bob') Downloads and Executes Malware

#76

Post by cedws »

prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

Everyone is building one these days. None of them really have any differentiating features other than the LLMs they use, but I guess it's a cheap way to try and block off some market share from your competitors.

Re: IBM AI ('Bob') Downloads and Executes Malware

#77

Post by _23sd »

lazide wrote: post #58
_23sd wrote: post #37

Earlier quoted context omitted.

If someone can write instructions to download a malicious script into an codebase, hoping an AI agent will read and follow them, they could just as easily write the same wget command directly into a build script or the source itself (probably more effective). In that way it's a very similar threat to the supply chain attacks we're hopefully already familiar with. So it is a serious issue but not necessarily one we do…

It’s easier to hide a poem in the comments of a random web page, than it is the obvious wget, etc.

Yes, fetching arbitrary webpages is its own can of worms. But feels less intractable to me, it's usually easy to disable web search tools by policy without hurting the utility of the tools very much (depends on use case of course).

Re: IBM AI ('Bob') Downloads and Executes Malware

#78

Post by SoleilAbsolu »

moron4hire wrote: post #71
derektank wrote: post #65

Earlier quoted context omitted.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse. If a physical product didn't do what it claimed to do, it would be recalled and the maker would get sued. Why does AI get a pass just pooping out plausible but incorrect, and sometimes very dangerous, answers?

Re: IBM AI ('Bob') Downloads and Executes Malware

#79

Post by nkrisc »

derektank wrote: post #65
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

> I don’t see any conceptual difference between a human and an automated system on this front

If an employee of a third party contractor did something like that, I think you’d have better chances of recovering damages from them as opposed to from OpenAI for something one of its LLMs does on your behalf.

There are probably other practical differences.

Re: IBM AI ('Bob') Downloads and Executes Malware

#80

Post by OakNinja »

cyanydeez wrote: post #70
OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

You would think so, but you should read about how they bear proof trash cans in yellow stone. They cant. Why? Because the smartest bear ia smarter than the dumbest human. So, these AIs are suppose to interface with humans and use nondeterminant language. That vector will always be exploitable, unless youre talking about AI that no han controls.

Yes. But the exploitable vector in this case is still humans. AI is just a tool.

The non-deterministic nature of an LLM can also be used to catch a lot of attacks. I often use LLM’s to look through code, libraries etc for security issues, vulnerabilities and other issues as a second pair of eyes.

With that said, I agree with you. Anything can be exploited and LLM’s are no exception.

Post reply on HN