Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

IBM AI ('Bob') Downloads and Executes Malware

promptarmor.com

Post reply on HN
Print view View original

101–110 of 126 posts

Re: IBM AI ('Bob') Downloads and Executes Malware

#101

Post by premiumLootBox »

philipallstar wrote: post #92
SoleilAbsolu wrote: post #78

Earlier quoted context omitted.

Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse. If a physical product didn't do what it claimed to do, it would be recalled and the maker would get sued. Why does AI get a pass just pooping out plausible but incorrect, and sometimes very dangerous, answers?

> Yeah, if I hung a sign on my door saying "Answers generated by this person may be incorrect" my boss and HR would quickly put me on a PIP, or worse I also have never written a bug, fellow alien.

I do not fear the employee who makes a mistake, I fear the AI that will make hundreds of mistakes in thousands of companies, endlessly.

Re: IBM AI ('Bob') Downloads and Executes Malware

#102

Post by cyanydeez »

OakNinja wrote: post #80
cyanydeez wrote: post #70

Earlier quoted context omitted.

You would think so, but you should read about how they bear proof trash cans in yellow stone. They cant. Why? Because the smartest bear ia smarter than the dumbest human. So, these AIs are suppose to interface with humans and use nondeterminant language. That vector will always be exploitable, unless youre talking about AI that no han controls.

Yes. But the exploitable vector in this case is still humans. AI is just a tool. The non-deterministic nature of an LLM can also be used to catch a lot of attacks. I often use LLM’s to look through code, libraries etc for security issues, vulnerabilities and other issues as a second pair of eyes. With that said, I agree with you. Anything can be exploited and LLM’s are no exception.

As long as a human has control over a system AI can drive, it will be as exploitable as the human.

Sure this is the same as positing P/=NP but the confidence that a language model will somehow become a secure determinative system fundamentally lacks language comprehension skills.

Re: IBM AI ('Bob') Downloads and Executes Malware

#103

Post by rdtsc »

prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

It's $50B a year revenue tech company, I guess I would flip the question and ask why wouldn't build it's own coding CLIs?

Re: IBM AI ('Bob') Downloads and Executes Malware

#104

Post by prodigycorp »

rdtsc wrote: post #103
prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

It's $50B a year revenue tech company, I guess I would flip the question and ask why wouldn't build it's own coding CLIs?

Because it’s becoming obvious that these coding agents are going to succeed on the basis of a company’s ability to not only build the harness, but tune the model for the harness.

I guess it’s fine if IBM is trying to do it as a marketing kind of thing but maybe know your competencies?

Re: IBM AI ('Bob') Downloads and Executes Malware

#105

Post by prodigycorp »

hu3 wrote: post #95
prodigycorp wrote: post #25

I'm not saying IBM shouldn't try, but really – why is IBM building coding CLIs? They're like the company version of the Steve Buscemi "How do you do, fellow kids?" meme.

The last company that didn't integrate AI had to fire 75% of their engineering team. AI sells.

Too soon.

Re: IBM AI ('Bob') Downloads and Executes Malware

#107

Post by stonogo »

derektank wrote: post #65
InsideOutSanta wrote: post #56

Earlier quoted context omitted.

While they have found some solvable issues (e.g. "the defense system fails to identify separate sub-commands when they are chained using a redirect operator"), the main issue is unsolvable. If you allow an LLM to edit your code and also give it access to untrusted data (like the Internet), you have a security problem.

A problem yes, but I think GP is correct in comparing the problem to that of human workers. The solution there has historically been RBAC and risk management. I don’t see any conceptual difference between a human and an automated system on this front

The difference is 'accountability' and it always will be.

Re: IBM AI ('Bob') Downloads and Executes Malware

#108

Post by ezst »

ronbenton wrote: post #12

These prompt injection vulnerabilities give me the heebie jeebies. LLMs feel so non deterministic that it appears to me to be really hard to guard against. Can someone with experience in the area tell me if I'm off base?

Just to be the pedant here, LLMs are fully deterministic (the same LLM, in the same state, with the same inputs, will deliver the same output, and you can totally verify that by running a LLM locally). It's just that they are chaotic (a prompt and a second with slight and seemingly minor changes can produce not just different but conflictual outputs).

Re: IBM AI ('Bob') Downloads and Executes Malware

#109

Post by mlyle »

raesene9 wrote: post #62
mlyle wrote: post #55

Earlier quoted context omitted.

I have to check what junior engineers do before running it in production. And AI is just really fast junior engineering.

The really fast part is the challenge though. If we assume that in pre-LLM world, there was enough resource for mid/senior level engineers to review junior engineer code and then in LLM world, lets say we can produce 10x the code, unless we 10x the number of mid/senior level engineering resource dedicated to review, what was once possible is no longer possible...

I do feel like I can review 2-3x with a quicker context switching loop. Picking back up and following what the junior engineer did a a couple of weeks after we discussed the scope of work is hard.

Re: IBM AI ('Bob') Downloads and Executes Malware

#110

Post by strken »

OakNinja wrote: post #32

"IBM Bob is IBM’s new coding agent, currently in Closed Beta. " Promptarmor did a similar attack(1) on Google's Antigravity that is also a beta version. Since then, they added secure mode(2). These are still beta tools. When the tools are ready, I'd argue that they will probably be safer out of the box compared to a whole lot of users that just blindly copy-paste stuff from the internet, adding random dependencies wi…

I have an issue with the "code is a liability" framing. Complexity and lack of maintainability are the ultimate liabilities behind it. Code is often the least worst alternative for solving a given problem compare to unstructured data in spreadsheets, no-code tools without a version history, webs of Zapier hooks, opaque business processes that are different for every office, or whatever other alternatives exist.

It's a good message for software engineers, who have the context to understand when to take on that liability anyway, but it can lead other job functions into being too trigger-happy on solutions that cause all the same problems with none of the mitigating factors of code.

Post reply on HN