How a Texas student blew the whistle on a rogue AI hacking attempt • Hacker News Board

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#111

Post by chrisjj » Sun, Aug 23, 2026, 3:23 PM UTC

Kim_Bruning wrote: post #90

Kim_Bruning wrote: post #88

Earlier quoted context omitted.

> You've been fooled by a next-token predictor. I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates? That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no…

I'm going to add this as a separate comment: These kinds of stories probably read very differently for someone who uses Opus and Fable agents all day and goes "ohhh, I saw this in miniature last week; this and this and this must have happened" , vs someone who tried free-tier Gemini flash one rainy Sunday, got hallucinated at, and concludes it must all be a scam.

Probably everything reads very differently to someone who talks to chatbots all day.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#112

Post by jpc0 » Sun, Aug 23, 2026, 3:37 PM UTC

Kim_Bruning wrote: post #102

jpc0 wrote: post #96

Earlier quoted context omitted.

My argument isn’t against those that actually put in the effort and got held accountable, it’s against the “we gave our agent bash and internet and it hacked xyz”. Bash and internet in that example might be highly abstracted but it’s still bash and internet. Just look at the replies in this very comment thread, it’s pretty much “We tried nothing and we’re all out of ideas” In the only other discipline you mentioned,…

Ah , well, on HN you ARE supposed to go for the steel-man. And the steel-man happens to be closer to reality here, more like: "We gave our agent a harness and put it inside a test environment and told it to keep hacking at an objective within that environment until it solved it." 'cept it turned out the container environment had a few flaws -which it always will- and the agent deemed it easier to escape out and try a…

My argument there is likely there was too much surface area in the environmental to start with.

If your webserver bundled the kitchen sink but you never used it the easiest way to make it more secure is to remove the kitchen sink from production code/codepaths.

There may very well be legitimate edge cases where there is some novel issue found but in some of these cases the AI had arbitrary web access when all the task required was very specific web access, we’ve been able to parse urls for a very long time and it is rather trivial to just deny a toolcall if it is outside of the expected domain.

But that’s the hard way that requires time and diligence to do, the easy way is give it access to curl and ask it to not do anything bad while setting up its only feedback to be to solve the problem at hand.

We are really in an age where there are many exploits being found and patched, if an AI made use of a novel exploit then great, write up a report, patch/report the bug and apologise.

But using a case of clear engineering failure, and yes even if the failure is despite your best efforts, for marketing really does not seem like you have any intent to correct the issue.

And we can loop all the way back to regulation of AI, if the industry refuses to be better then governmental will do it instead and their solution will very likely be inferior in all ways.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#113

Post by exogenousdata » Sun, Aug 23, 2026, 3:57 PM UTC

blagie wrote: post #94

CoastalCoder wrote: post #63

Earlier quoted context omitted.

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

That's not quite what it says. "A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed." 1. The reason is well-regulated militias, but the right is of the people. 2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free. Now, here again, "well regulated" give…

"I don't think one could argue the militia could be under state control"

I'm not certain this is true at all. To suggest that the Founders meant for state militias to simply be their own forces with no control by the federal government is in direct conflict with the Articles of the US Constitution.

The US Constitution clearly outlines the powers of Congress to call forth & organize the militia. The US Constitution also clearly identifies the President as Command in Chief of the militia. That was further codified in a handful of acts in the 1790s, upheld by the Supreme Court in the early 1800s. The US Constitution also makes mention of the militia in the 5th Amendment.

Early writing at that time suggests that the reason some of the Founders supported state militias was because they were very reluctant to allow the US to maintain a standing army. The US Industry Military Complex was never intended by the Founders.

Today we identify the militia described in the US Constitution as the US Army Reserves. However that came about only after the passage of the Dick Act of 1903 (yes, that's actually the name) because President (Teddy) Roosevelt was upset at the state of the militias during the Spanish American War of 1898. And the Dick Act actually split the idea of a militia into 'organized' and 'unorganized'.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#114

Post by themaninthedark » Sun, Aug 23, 2026, 3:58 PM UTC

m4rtink wrote: post #89

ben_w wrote: post #82

Earlier quoted context omitted.

> Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware? Before LLMs got good enough to do this, lots of people were dismissive of their capabilities and didn't take seriously the idea that this was a risk to protect against. Then again, before LLMs, people were saying that obviously nobody would be dumb enough to put an AI on the internet where it could hack anyone,…

Anti radiolation missiles don't just launch automatically in most scenarios, not to mention discriminate quite a lot what they lock on to avoid simple jamming. Not to mention the AA radars they usually target being more powerful by orders of magnitude than a handheld radar gun.

I was in high-school when the war in Afghanistan started. The terrain in my area was mountainous so there were often low flying training flights... I always thought it would be cool to build a radar and ping one of the aircraft, especially wanted to know if it would fire countermeasures. But I didn't like the very high likelihood of an FBI investigation with possible terrorist charges.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#115

Post by exogenousdata » Sun, Aug 23, 2026, 4:00 PM UTC

vel0city wrote: post #100

CoastalCoder wrote: post #63

Earlier quoted context omitted.

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

Its not just implying the right is for the people, it's directly stated. It's "the right of the people to keep and bear arms". It doesn't say "the right of the militias" to keep and bear arms".

Then why was the militia mentioned at all?

For example, the 1st Amendment does not attempt to lay out some non-exclusive examples of why the rights in the 1st Amendment are included. So why did the Founders include this in the Amendment wording?

I think ignoring phrases in the US Constitution to fit a narrative without any consideration isn't a recipe for good governance. But I'm happy to be proven wrong.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#116

Post by chrisjj » Sun, Aug 23, 2026, 4:11 PM UTC

wongarsu wrote: post #107

chrisjj wrote: post #95

Earlier quoted context omitted.

> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. "so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up . Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its crea…

LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". The learning part is debatable, as is the level of success achieved The mash-up of the entire internet is the mechanism by which they attempt to achieve the goal, not the goal itself. And it's only the first training step

> LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason".

I think you've mistaken the sales pitch for the design. Not even the enclopedia anyone can edit comes remotely near that:

"A large language model (LLM) is an AI model (typically a neural network) trained on a vast amount of text for natural language processing tasks, especially language generation. LLMs can typically generate, summarize, translate, and analyze text in many contexts.[1] They are the basis for many modern chatbots, such as ChatGPT, Claude, Gemini, Grok, and DeepSeek.

LLMs are typically based on transformer architecture.[2] Generative pre-trained transformers (GPTs) are a type of LLM that is pre-trained to predict the next word.[3] GPTs are then often fine-tuned to follow instructions and to behave as assistants.[4]

Biased or inaccurate training data can make an LLM's output less reliable. Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety."

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#117

Post by Kim_Bruning » Sun, Aug 23, 2026, 4:26 PM UTC

chrisjj wrote: post #111

Kim_Bruning wrote: post #90

Earlier quoted context omitted.

I'm going to add this as a separate comment: These kinds of stories probably read very differently for someone who uses Opus and Fable agents all day and goes "ohhh, I saw this in miniature last week; this and this and this must have happened" , vs someone who tried free-tier Gemini flash one rainy Sunday, got hallucinated at, and concludes it must all be a scam.

Probably everything reads very differently to someone who talks to chatbots all day.

> Probably everything reads very differently to someone who talks to chatbots all day.

A chatbot is a particular kind of harness. Typically an LLM driving a chatbot won't be able to hack very much.

So we agree, someone who talks to bad chatbots all day probably has a very different view of SOTA agents. :-P

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#118

Post by chrisjj » Sun, Aug 23, 2026, 4:29 PM UTC

wongarsu wrote: post #104

chrisjj wrote: post #91

Earlier quoted context omitted.

> This is all without mentioning the fact that articles with drivel like "the AI messed up and then lied about it" implies a reasoning ability Moreover this implies, actually requires, intent to deceive - which these so-called AIs do not and cannot have. Their only "intent" is to maximise the credibility of their output.

AIs can set and work towards goals. Whether that is intent or just tokens and tool calls simulating an agent with intent seems like a distinction with no actionable difference

Here's the difference under discussion:

"When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake"

Honest, note.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#119

Post by Kim_Bruning » Sun, Aug 23, 2026, 4:30 PM UTC

chrisjj wrote: post #116

wongarsu wrote: post #107

Earlier quoted context omitted.

LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". The learning part is debatable, as is the level of success achieved The mash-up of the entire internet is the mechanism by which they attempt to achieve the goal, not the goal itself. And it's only the first training step

> LLMs are very explicitly designed to "understand, and make judgments or have opinions that are based on reason". I think you've mistaken the sales pitch for the design. Not even the enclopedia anyone can edit comes remotely near that: "A large language model (LLM) is an AI model (typically a neural network) trained on a vast amount of text for natural language processing tasks, especially language generation. LLMs…

>Benchmark evaluations for LLMs attempt to measure model reasoning, factual accuracy, alignment, and safety.

And you left out the refs.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#120

Post by ben_w » Sun, Aug 23, 2026, 5:11 PM UTC

chrisjj wrote: post #93

ben_w wrote: post #86

Earlier quoted context omitted.

Squirrels have been observed performing deception against other squirrels. Dis/honesty certainly requires some intelligence to pass, but it is a low bar, and one which research has shown that LLMs can perform, e.g. this paper linked from another comment in this discussion: https://arxiv.org/pdf/2509.03518

Paper says "These scenarios underscore a crucial challenge in AI safety: ensuring that LLMs were truthful in the first place." Hard to take seriously any research based on the premise that LLMs were truthful in the first placr. These chatbots have no understanding of truth. They simply parrot their inputs. Where fed falsehoods, they will output falsehoods - with a sprinkling of added fabrications euphemistically excu…

> These chatbots have no understanding of truth.

Sometimes I forget that for all that my philosophy qualification is mediocre, it is more than most people ever bother with.

Outside mathematics (and, I guess, "common sense" definitions that fail under the slightest scrutiny, scrutiny that normal people never bother to give), there is no agreement on "truth", there is only degree of belief and justification for that belief that itself terminates in one of three unsatisfactory ways:

https://en.wikipedia.org/wiki/I_know_that_I_know_nothing

https://en.wikipedia.org/wiki/Theories_of_truth

https://en.wikipedia.org/wiki/Münchhausen_trilemma

> Where fed falsehoods, they will output falsehoods - with a sprinkling of added fabrications euphemistically excused as "hallucinations".

Tu quoque. Which would be a fallacious charge if the point were not that "truth" is so hard to define, and that the reason you give for dismissing AI is something that applies to all.

(Hallucinations are not excused, they are a failure to be worked around).