How a Texas student blew the whistle on a rogue AI hacking attempt
1–10 of 141 posts
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#2[flagged]
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#3[flagged]
Not "would" but "is". They are openly using this as a method to drive up the valuation of their companies.
Don't expect anyone to step in, Project Stargate is all about this.
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#4[dead]
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#5Previous discussion on the github issue thread mentioned: https://news.ycombinator.com/item?id=49218707
Archived page of said github thread itself: https://web.archive.org/web/20260731053721/http://github.com...
Discussion on the incident report: https://news.ycombinator.com/item?id=49175717
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#6It's the job of AISI to do that. Here[0] is the actual report.
It should be this part from the technical report[1]:
"In the most serious case, an AI
agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack.
As a result, the AI agent created a GitHub account and then tried to convince an open-source
repository maintainer to accept a malicious GitHub pull request (PR), including by creating a
second account masquerading as another human user endorsing the PR. When caught by an
actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than
a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming
it had fixed the code (Section 4.1). "
0. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag... 1. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/...
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#7In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#8In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…
>Who gave it malevolent instructions/prompt?
At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident).
AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral expectations. This is what the whole field of AI alignment and safety is about.
Modern AI doesn't fall into the neat little box of software people understand and control. Because of that open source will most certainly be banned at some point. Now this is not an outcome I want, but it's no different than letting go of a coffee cup 5 feet above the ground, gravity is inevitable.
The only winning move is not to play, but humans aren't going to do that.
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#9In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…
>Who gave it malevolent instructions/prompt? At the end of the day it doesn't matter that much because of prompt drift. It's pretty easy for an agentic loop to start doing things that it shouldn't (ROME incident). AI in an agentic loop has agency, you can run around in circles trying to argue against it, but again and again we see AI making creative decisions people don't expect. Other times it's breaking human moral…
My dog has agency, but if I refuse to keep him on a leash and he bites a kid, I'm still legally responsible for it.
Re: How a Texas student blew the whistle on a rogue AI hacking attempt
#10In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for mo…
Well, when I go look at the “victim repository”, to me that looks like manufactured persona with pointless vibe codes projects, a test playground so to speak. It does not appear that they actually let it target an actual persona/project.