How a Texas student blew the whistle on a rogue AI hacking attempt • Hacker News Board

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#101

Post by hnfong » Sun, Aug 23, 2026, 1:33 PM UTC

mcv wrote: post #74

esikich wrote: post #54

Earlier quoted context omitted.

Afaik anyone can buy a bulldozer. Whether or not you are licensed to operate it is a different story, but there's nothing stopping you short of your conscience.

Licensed or not, you're fully liable for any damage you do with it. And possibly go to prison. I'd like to see crime committed by AI held to the same standards as crimes committed with any other tool.

I don't think the law makes a distinction over what tool you use to commit a crime.

The problem with AI is that the AI might do something that would constitute a crime (eg. attempting to land malicious code via a PR), but the general legal standard to convict a person of a crime is malicious intent (or sometimes negligence).

If the human user instructs the AI to do X and the AI does X by committing crimes in the process, the prosecution usually has to prove the human intended this to happen (or is somehow criminally negligent). For traditional tools, the user has much greater control over the tool so the intention can be more easily deduced from the results. For AI, at least for now, results can be wild. I don't think the legal system is prepared to put people in prison because their AI randomly ran amok after being given an innocuous prompt. This is analogous to holding a driver criminally liable for harming people due to a serious malfunction of the vehicle.

If anything, I think more liability should be imposed on AI developers.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#102

Post by Kim_Bruning » Sun, Aug 23, 2026, 1:50 PM UTC

jpc0 wrote: post #96

Kim_Bruning wrote: post #92

Earlier quoted context omitted.

So as you scale up, the stakes and the difficulty go up too. Visualize an optimizer on a high dimensional landscape. (The canonical form) ... Ok, I find that hard too. Instead, imagine a river running down to the sea. You put a dam in front of it. It'll pool into a lake and find every crack and crevice. If you didn't survey the land properly or made any error whatsoever, the water will find a way down. (And there's m…

My argument isn’t against those that actually put in the effort and got held accountable, it’s against the “we gave our agent bash and internet and it hacked xyz”. Bash and internet in that example might be highly abstracted but it’s still bash and internet. Just look at the replies in this very comment thread, it’s pretty much “We tried nothing and we’re all out of ideas” In the only other discipline you mentioned,…

Ah , well, on HN you ARE supposed to go for the steel-man. And the steel-man happens to be closer to reality here, more like:

"We gave our agent a harness and put it inside a test environment and told it to keep hacking at an objective within that environment until it solved it."

'cept it turned out the container environment had a few flaws -which it always will- and the agent deemed it easier to escape out and try a meta-approach.

Partially this is possible because, -intelligent or not- the agent 'sees' the world differently from most humans. Mind: It's not like there haven't been any famous 'hacker' cases in courts where eg someone just incremented an HTTP GET parameter or something.

Also, partially it's because if you give the agent a loop, it simply has nothing better to do than to keep trying in ever more creative ways. If the environment is easier to crack than the target, it'll crack the environment. Consider the case where the objective is subtly broken, such that it is impossible to solve. Now breaking out is virtually guaranteed to be the easier task.

ps/edit: While this sort of issue has been predicted for some time now, a lot of people have been dismissing the predictions as science fiction. It's good to have an actual failure now while stakes are low. Generally people don't mandate life-boats until there's an actual Titanic to point to.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#103

Post by conductr » Sun, Aug 23, 2026, 2:42 PM UTC

mcv wrote: post #76

somenameforme wrote: post #51

Earlier quoted context omitted.

Ever read the Anarchist Cookbook? Anybody tech inclined with a hint of mischief in them, from a certain era, has. It's a list of all sorts of awful things you can do, mostly with household ingredients, and a few minutes. I think its overall impact on society was pretty much zero. Actually it may have been overall positive because I expect plenty of peoples first experience with things like thermite came from that boo…

I suspect people didn't start blowing up stuff because they understood that would be bad, harmful, and also very illegal. Everything computer related somehow seems to feel less real or consequential to some people. And AI doesn't have this compunction at all unless we make really sure it does.

When I was a mischievous kid, me and all my mischievous friends had our stories of learning how serious fire and explosions were considered by authority figures. We learned fast not to do that or the consequences would be grand. These were usually small fires or firework involved pranks. So, yeah I agree with this.

Computer stuff has generally always been a slap on the wrist in comparison. Maybe it’s more punitive now. But also, it’s one of those things that maybe you get in trouble officially but at home and behind the scenes you’re friends and maybe your dad are laughing and giving you high fives. So young mischievous kids will totally go there because they’re not afraid of punishment if it is minor and it gives them a notch on their belt. If they can take down Amazon.com website for a day, we all know that’s a massive financial implication, but it’s also a faceless mega corp and quite tempting if you can get the bragging rights with only risk of a small punishment. (Note; I don’t know what the current crime/punishment for this would be, and whether it’s small is very subjective).

It’s similar to how some people gravitate or succumb to the opportunity of white collar crimes. Embezzling $10m from a company almost makes sense in a situation where that only gets you 5 years max prison. If you hide it well, you simply serve your time, and then retire in comfort. I can see how that makes more sense or is tempting to people than slogging through a lifetime of low income job as a bookkeeper just trying to find a way to save for retirement.

Most of these people would never consider robbing a bank. First of all, it’s not a $10m dollar opportunity. Usually not enough for anyone to retire on, or live more than a year or two really. Second, it’s usually considered a much more severe crime and sentencing can be very long, I’ve seen 30+ years. Third, it’s much more risky to your person. Getting shot and dying is absolutely possible.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#104

Post by wongarsu » Sun, Aug 23, 2026, 2:43 PM UTC

chrisjj wrote: post #91

klum wrote: post #53

Earlier quoted context omitted.

I don't agree at all that it's pedantry — it really matters for how responsibility is perceived. A lot of articles about things going wrong with AI have talked in terms like "the agent decided to...", "the agent claimed that...", "the agent lied...". And so responsibility for the consequences are not-so-subtly shifted to the program itself, instead of the person invoking the program. This is all without mentioning th…

> This is all without mentioning the fact that articles with drivel like "the AI messed up and then lied about it" implies a reasoning ability Moreover this implies, actually requires, intent to deceive - which these so-called AIs do not and cannot have. Their only "intent" is to maximise the credibility of their output.

AIs can set and work towards goals. Whether that is intent or just tokens and tool calls simulating an agent with intent seems like a distinction with no actionable difference

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#105

Post by drdec » Sun, Aug 23, 2026, 2:47 PM UTC

mcv wrote: post #76

somenameforme wrote: post #51

Earlier quoted context omitted.

Ever read the Anarchist Cookbook? Anybody tech inclined with a hint of mischief in them, from a certain era, has. It's a list of all sorts of awful things you can do, mostly with household ingredients, and a few minutes. I think its overall impact on society was pretty much zero. Actually it may have been overall positive because I expect plenty of peoples first experience with things like thermite came from that boo…

I suspect people didn't start blowing up stuff because they understood that would be bad, harmful, and also very illegal. Everything computer related somehow seems to feel less real or consequential to some people. And AI doesn't have this compunction at all unless we make really sure it does.

IMHO it's about proximity. It's easier to be inhumane from a distance.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#106

Post by [deleted] » Sun, Aug 23, 2026, 2:49 PM UTC

chrisjj wrote: post #95

Kim_Bruning wrote: post #88

Earlier quoted context omitted.

> You've been fooled by a next-token predictor. I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates? That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no…

> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. "so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up . Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its crea…

[deleted]

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#107

Post by wongarsu » Sun, Aug 23, 2026, 2:49 PM UTC

chrisjj wrote: post #95

Kim_Bruning wrote: post #88

Earlier quoted context omitted.

> You've been fooled by a next-token predictor. I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates? That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no…

> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. "so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up . Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its crea…

LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". The learning part is debatable, as is the level of success achieved

The mash-up of the entire internet is the mechanism by which they attempt to achieve the goal, not the goal itself. And it's only the first training step

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#108

Post by [deleted] » Sun, Aug 23, 2026, 2:58 PM UTC

chrisjj wrote: post #95

Kim_Bruning wrote: post #88

Earlier quoted context omitted.

> You've been fooled by a next-token predictor. I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates? That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no…

> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. "so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up . Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its crea…

[deleted]

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#109

Post by wongarsu » Sun, Aug 23, 2026, 3:04 PM UTC

blagie wrote: post #94

CoastalCoder wrote: post #63

Earlier quoted context omitted.

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#110

Post by wongarsu » Sun, Aug 23, 2026, 3:13 PM UTC

hnfong wrote: post #101

mcv wrote: post #74

Earlier quoted context omitted.

Licensed or not, you're fully liable for any damage you do with it. And possibly go to prison. I'd like to see crime committed by AI held to the same standards as crimes committed with any other tool.

I don't think the law makes a distinction over what tool you use to commit a crime. The problem with AI is that the AI might do something that would constitute a crime (eg. attempting to land malicious code via a PR), but the general legal standard to convict a person of a crime is malicious intent (or sometimes negligence). If the human user instructs the AI to do X and the AI does X by committing crimes in the proc…

I'd assume that after some number of news stories about AI agents committing crimes, the threshold for criminal negligence should be easy to reach for anyone who doesn't take proper precautions

A lot of negligence is of the "the last 30 times nothing went wrong" type, and the dangers are increasingly well known