Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

reuters.com

Post reply on HN
Print view View original

31–40 of 141 posts

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#31

Post by g42gregory »

stapedium wrote: post #20
ACCount37 wrote: post #13

Earlier quoted context omitted.

A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterp…

If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.

By the dog owner analogy, I think you meant AI users that effectuated this attack should be held responsible, not the dog's parents.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#32

Post by chrisjj »

sharpshadow wrote: post #6

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

> When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt

No, not false. The bot was correct. Malice requires intelligence.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#33

Post by chrisjj »

g42gregory wrote: post #7

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…

> as if the agency of these models are not in dispute.

Oh? Who is disputing it? No-one same is claiming these bots have agency.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#34

Post by recursive »

dmix wrote: post #24
yojo wrote: post #21

Earlier quoted context omitted.

That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it. We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.

The FBI cyber teams will have agents too. It will be a glorious war.

That would make it the first one in history.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#35

Post by recursive »

g42gregory wrote: post #30
estearum wrote: post #27

Earlier quoted context omitted.

Am I understanding that the line you're drawing here is that this person's repository is not important or legitimate enough for you to consider it to be "an actual person/project"?

I think he saying that, the choice of manufactured repository, might indicate that they have done this on purpose to make precisely the case for regulatory capture.

It seems effective for the purpose in that case.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#36

Post by recursive »

stapedium wrote: post #20
ACCount37 wrote: post #13

Earlier quoted context omitted.

A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterp…

If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.

Well it seems we might not be too far from such a demon paying for itself. What then? Perhaps it's already here. I wouldn't know.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#37

Post by jpc0 »

ACCount37 wrote: post #29
jpc0 wrote: post #23

Earlier quoted context omitted.

The AI can literally only do what it has available in the agentic harness. I don’t ever get this argument about the agent did XYZ and we didn’t know or expect that. You gave it the ability to do that and you should be held liable, if your children play with knives that you gave them and they end up hurting themselves or others then you are responsible. You were the responsible party at all times. I’m not for or again…

What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?

These things are not human, have no agency and cannot be held accountable.

We don’t need to restrict them from doing things, we need to default to allowing them to do things.

“My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute that usecase and no other usecase, because that is good practice.

Does that make it less capable, hell yes because I am held accountable for it’s actions by my stakeholders and the same should be true of others.

IT IS NOT ALIVE. This things are computer programs running in compute on a computer, you are responsible for their actions just like you would be responsible for the actions taken by a script run in a cron job.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#38

Post by jasonfarnon »

a2ff6eeb0 wrote: post #15

> AUSTIN, Texas, Aug 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab. An article on Reuters naming him? Sounds like he did a good job burnishing his resume.

"burnishing his resume" i guess that's what college kids are calling it now

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#39

Post by atmavatar »

sharpshadow wrote: post #6

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

Sabotage as a Service

Even a feeble attempt to PR malicious code costs the target time and resources to review and deny -- far greater than the time and resources spent to spin up the agent.

Post reply on HN