Live data from Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

reuters.com

51–60 of 141 posts

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#51
post #21

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…

That’s nice in theory, but as these things get better and cheaper this kind of capability is going to drop from nation states to script kiddies. That future is coming, I don’t see any way around it. We can round up all the bored teenagers we want, but it’s not putting the genie back. Better start adjusting our systems to account for it.

Ever read the Anarchist Cookbook? Anybody tech inclined with a hint of mischief in them, from a certain era, has. It's a list of all sorts of awful things you can do, mostly with household ingredients, and a few minutes. I think its overall impact on society was pretty much zero. Actually it may have been overall positive because I expect plenty of peoples first experience with things like thermite came from that book, and now there are all sorts of videos and neat experiments with such on sites like YouTube.

I think this is in part because most people, including awful, tend to be relatively morally inclined. But I also think because even with an LLM, doing things takes effort. And if you're willing to dedicate effort towards a task, there tend to be way more rewarding/gratifying things to do than try to hurt people. Countries tend to be excessively sociopathic because you have large scale 'intelligence' organizations who see their entire point of existence as being to engage in misdeeds.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#52
post #32

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

> When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt No, not false. The bot was correct. Malice requires intelligence.

An "honest mistake" requires the same amount of intelligence as malice. What weird pedantry.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#53
post #32

Earlier quoted context omitted.

> When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt No, not false. The bot was correct. Malice requires intelligence.

Nobody was confused or misled by what was written. We all understand what is meant. I can’t even call this pedantry—it’s just you asking everyone to subscribe to your particular desired style of talking about this stuff.

I don't agree at all that it's pedantry — it really matters for how responsibility is perceived. A lot of articles about things going wrong with AI have talked in terms like "the agent decided to...", "the agent claimed that...", "the agent lied...". And so responsibility for the consequences are not-so-subtly shifted to the program itself, instead of the person invoking the program.

This is all without mentioning the fact that articles with drivel like "the AI messed up and then lied about it" implies a reasoning ability which, as far as I understand, is not there at all. But writing this way shapes people's perception of how "AI" works.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#54
post #43

Earlier quoted context omitted.

People have caused lots of damage with bulldozers.

If your point is that we should regulate AI as least as strictly as industrial vehicles, I agree.

Afaik anyone can buy a bulldozer. Whether or not you are licensed to operate it is a different story, but there's nothing stopping you short of your conscience.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#55
post #14

In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…

>Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents. But some tools (guns) are regulated.

Personally, I think gun companies should be liable for any harm done by their products as well.

We want rule-of-law, and in the US, people should have an absolute right to bare arms, as in the second amendment. Free market forces can then determine appropriate prices, insurance, and protective measures to make sure those guns are managed safely.

If I want an F35 and an Abrams, that's okay, so long as Lockheed and General Dynamics are willing to sign off (with full liability for damages) that I'm managing them safely.

Free markets work pretty well with:

a) Full transparency, as needed for rational decision-making

b) No way to externalize costs

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#56
post #37

Earlier quoted context omitted.

What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?

These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute tha…

Accountability is worthless, and always was. AIs just show it plain for everyone to see.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#57

Earlier quoted context omitted.

A lot of people somehow seem to think that the user prompt is the be-all and end-all of AI behavior. Prompts aren't code. They are instructions. Orders given to an eager and somewhat demented demon. The prompt can easily "wash out" of the demon's working memory by the end of a session. The demon can get sidetracked by some subgoal and never get back on track. The instruction can get misinterpreted, and that misinterp…

If the user input can’t control the demon, then the person or company feeding the demon (ie paying the electric bill and collecting $$$ from users) is responsible. At the end of the day, dogs and cars are the same as data centers. If your dog bites by kid or your car rolls down the hill and hits my house, you are responsible for the damage. AI providers should be held to the same standard.

The user input can control the demon most of the way, most of the time!

We don't know how to obtain full, absolute, guaranteed control over a demon while still having a useful demon. Might be impossible. Forbidden knowledge be like that - it's not the best thing if you want your life to be full of certainties.

But the demons are very useful. And they're getting more useful still. So we aren't about to stop.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#59
post #32

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

> When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt No, not false. The bot was correct. Malice requires intelligence.

So does honesty. So it was still a false claim.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#60

It's the job of AISI to do that. Here[0] is the actual report. It should be this part from the technical report[1]: "In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by c…

Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware?

Should weapon manufacturers test their weapons by starting wars?

I would expect more responsibility from a government agency.

Post reply on HN