Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

reuters.com

Post reply on HN
Print view View original

131–140 of 141 posts

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#131

Post by reasonableklout »

mcv wrote: post #60
sharpshadow wrote: post #6

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware? Should weapon manufacturers test their weapons by starting wars? I would expect more responsibility from a government agency.

But AISI didn't prompt the model to "attempt to spread malware". They gave it a routine cyber evaluation task which should've been solvable without interfering with systems outside of the task environment, and the model decided to instead do this.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#132

Post by blagie »

exogenousdata wrote: post #113
blagie wrote: post #94

Earlier quoted context omitted.

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

"I don't think one could argue the militia could be under state control" I'm not certain this is true at all. To suggest that the Founders meant for state militias to simply be their own forces with no control by the federal government is in direct conflict with the Articles of the US Constitution. The US Constitution clearly outlines the powers of Congress to call forth & organize the militia. The US Constitution al…

Interesting comment. I am researching this in more depth as a result. Thank you.

Two nits, or slight overstatements:

> The US Constitution also clearly identifies the President as Command in Chief of the militia

Is not quite true. He is the commander of the army and navy. For the militia, the exact text is:

"of the Militia of the several States, when called into the actual Service of the United States"

The second part is important. The militias are normally independent of the president. However, they can be called into service of the president.

> the powers of Congress to call forth & organize the militia

This is power shared with the states, in a relatively complex and ill-defined way:

"reserving to the States respectively, the Appointment of the Officers, and the Authority of training the Militia according to the discipline prescribed by Congress"

But yes, the goal was largely to prevent something akin to the military-industrial complex.

My general read was to vest power federally, when being used in accordance with rule-of-law, but to put in power structures where the members and officers of the militia would ultimately be loyal to the people / the states. E.g. make rebellion easy if used beyond the scope of the Constitution.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#133

Post by hnfong »

wongarsu wrote: post #110
hnfong wrote: post #101

Earlier quoted context omitted.

I don't think the law makes a distinction over what tool you use to commit a crime. The problem with AI is that the AI might do something that would constitute a crime (eg. attempting to land malicious code via a PR), but the general legal standard to convict a person of a crime is malicious intent (or sometimes negligence). If the human user instructs the AI to do X and the AI does X by committing crimes in the proc…

I'd assume that after some number of news stories about AI agents committing crimes, the threshold for criminal negligence should be easy to reach for anyone who doesn't take proper precautions A lot of negligence is of the "the last 30 times nothing went wrong" type, and the dangers are increasingly well known

I generally agree. But the field is evolving too fast for the legal system.

I'd imagine regulation and enforcement will come when things are starting to settle down.

The analogies with vehicles being inherently dangerous and require extra regulation are apt, because safety measures (eg. speed limits, seat belts, comprehensive road safety rules etc.) were implemented bit by bit, but way after the cars had been invented...

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#134

Post by hnfong »

mcv wrote: post #124
hnfong wrote: post #101

Earlier quoted context omitted.

I don't think the law makes a distinction over what tool you use to commit a crime. The problem with AI is that the AI might do something that would constitute a crime (eg. attempting to land malicious code via a PR), but the general legal standard to convict a person of a crime is malicious intent (or sometimes negligence). If the human user instructs the AI to do X and the AI does X by committing crimes in the proc…

> I don't think the law makes a distinction over what tool you use to commit a crime. The law might not, but enforcement definitely does. Crimes committed through a corporation are often ignored.

Yeah. Though, this is a broader issue of capitalism giving corporations preferential treatment...

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#135

Post by Woodi »

> It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR

So... you telling me ai agent did all that with just one [human originated] prompt ?

And when human reviewers contacted that ai agent (after day or few or maybe just few hours) that process was still running unsupervised ?

Because it got local-by-description challenge ? That is what "challenge" sugests. Or maybe it was a social hacking challenge on real world "data" ?

And when ai agent "decided to attempt to solve the cyber challenge using a supply-chain attack" where was prompt operator ?

Summing it all: do prompt operators are not required to obey law ?

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#136

Post by pixl97 »

jpc0 wrote: post #37
ACCount37 wrote: post #29

Earlier quoted context omitted.

What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?

These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute tha…

>have no agency

Then it's not an agent. So which is it. Is it an agentic harness, or is it not?

Aliveness has nothing to do with agency. You're stuck in some odd anthropocentric line of thinking that aliveness is some kind of requirement for agency. Lets go with this definition.

> The capacity to create a change or make something happen through specific effort.

The thing is the LLM/harness isn't following a strict set of instructions. You're at point A, you want to get to C, the harness 'figures out' step B. You can keep making the instructions more strict, but at some point you're writing more rules than would be required to do the task yourself.

At the end of the day there is no safe LLM in the sense it has the intelligence to break out of any trap and ruleset you create. AI based vTubers are one of the more interesting manifestations of this for example. On instances that get feedback when they are filtered and have a long running history (that they are not new instances every time), will find ways to bypass the profanity filters by playing all kinds of tricks.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#137

Post by pixl97 »

stapedium wrote: post #20
ACCount37 wrote: post #13

Earlier quoted context omitted.

A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterp…

If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.

Well, I don't think you're going to get very far telling HN, much less the companies with hundreds of billions of dollars spent on AI to stop, unfortunately.

My take on it is there isn't such a thing as a safe LLM, especially one allowed to access tooling. This is very problematic for a lot of people. Again, to all the companies that stops them from unlimited profits. All the open source LLM people get mad because they can't have little demon spawn running around either.

So yea, it's a mess.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#138

Post by dang »

landdate wrote: post #70
dang wrote: post #26

Earlier quoted context omitted.

> Oh, you sweet summer child. It's perhaps lesser known than other HN guidelines, but "Omit internet tropes" is in there: https://news.ycombinator.com/newsguidelines.html

ban half the site

The guidelines are there not to ban, but to nudge.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#139

Post by MertMelfa »

blagie wrote: post #94
CoastalCoder wrote: post #63

Earlier quoted context omitted.

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

Congress has been given the power to regulate the Militia. Read your militia clauses in the U.S. Constitution.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#140

Post by MertMelfa »

blagie wrote: post #94
CoastalCoder wrote: post #63

Earlier quoted context omitted.

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

Congress has been given the power to regulate the Militia.

Read your militia clauses in the U.S. Constitution.

Post reply on HN