Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How a Texas student blew the whistle on a rogue AI hacking attempt

reuters.com

Post reply on HN
Print view View original

91–100 of 141 posts

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#91

Post by chrisjj »

klum wrote: post #53
UqWBcuFx6NV4r wrote: post #45

Earlier quoted context omitted.

Nobody was confused or misled by what was written. We all understand what is meant. I can’t even call this pedantry—it’s just you asking everyone to subscribe to your particular desired style of talking about this stuff.

I don't agree at all that it's pedantry — it really matters for how responsibility is perceived. A lot of articles about things going wrong with AI have talked in terms like "the agent decided to...", "the agent claimed that...", "the agent lied...". And so responsibility for the consequences are not-so-subtly shifted to the program itself, instead of the person invoking the program. This is all without mentioning th…

> This is all without mentioning the fact that articles with drivel like "the AI messed up and then lied about it" implies a reasoning ability

Moreover this implies, actually requires, intent to deceive - which these so-called AIs do not and cannot have. Their only "intent" is to maximise the credibility of their output.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#92

Post by Kim_Bruning »

jpc0 wrote: post #37
ACCount37 wrote: post #29

Earlier quoted context omitted.

What's available in the agentic harness is: shell toolcall. That's just about every agentic harness, by the way. Good luck have fun. We have never solved "how do we restrict a user in a way that doesn't stop the user from doing useful things, but stops the user from doing harmful things" with humans either. Why do you expect AI to be any different?

These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute tha…

So as you scale up, the stakes and the difficulty go up too.

Visualize an optimizer on a high dimensional landscape. (The canonical form)

... Ok, I find that hard too.

Instead, imagine a river running down to the sea. You put a dam in front of it. It'll pool into a lake and find every crack and crevice. If you didn't survey the land properly or made any error whatsoever, the water will find a way down. (And there's many historic incidents where the dam even outright collapses)

For a more proximal approximation: lock treats in the kitchen cabinet in sight of little kids or kittens; then turn your back for Just One Gosh Darn Cotton Picking Moment(tm).

It seems the engineer who thinks their ship is unsinkable is the most likely to sink it. Are you sure your harness is as secure as you think it is? Will it stand up to ever more powerful models? Do you think engineers at eg Anthropic aren't at least as careful as you are?

(I've found that the 'only permitted actions' approach is not necessarily all that secure once deployed IRL)

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#93

Post by chrisjj »

ben_w wrote: post #86
mcv wrote: post #59

Earlier quoted context omitted.

So does honesty. So it was still a false claim.

Squirrels have been observed performing deception against other squirrels. Dis/honesty certainly requires some intelligence to pass, but it is a low bar, and one which research has shown that LLMs can perform, e.g. this paper linked from another comment in this discussion: https://arxiv.org/pdf/2509.03518

Paper says "These scenarios underscore a crucial challenge in AI safety: ensuring that LLMs were truthful in the first place."

Hard to take seriously any research based on the premise that LLMs were truthful in the first placr.

These chatbots have no understanding of truth. They simply parrot their inputs. Where fed falsehoods, they will output falsehoods - with a sprinkling of added fabrications euphemistically excused as "hallucinations".

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#94

Post by blagie »

CoastalCoder wrote: post #63
blagie wrote: post #55

Earlier quoted context omitted.

Personally, I think gun companies should be liable for any harm done by their products as well. We want rule-of-law, and in the US, people should have an absolute right to bare arms, as in the second amendment. Free market forces can then determine appropriate prices, insurance, and protective measures to make sure those guns are managed safely. If I want an F35 and an Abrams, that's okay, so long as Lockheed and Gen…

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

That's not quite what it says.

"A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed."

1. The reason is well-regulated militias, but the right is of the people.

2. The militia isn't a state apparatus. Indeed, the goal of the militia is to enable a rebellion if the state is no longer free.

Now, here again, "well regulated" gives plenty of leeway. For example, one might argue that the following scheme fits:

1. I can have whatever arms I want, including an F35

2. The F35 lives with a militia, which is well-regulated. I can use it in trainings there.

I don't think one could argue the militia could be under state control (that defeats the purpose!), but one could easily argue that it could be well-enough regulated that the current far-right extremist groups would not fit.

The concept was a group of citizens under e.g. a town / city council.

That's obviously not where case law went, but in an alternative reality, it very well might have.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#95

Post by chrisjj »

Kim_Bruning wrote: post #88
chrisjj wrote: post #77

Earlier quoted context omitted.

You've been fooled by a next-token predictor.

> You've been fooled by a next-token predictor. I also have a so called "pocket calculator" left over from when I went to school. Is this false? Have I been fooled by a little box of logic gates? That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up. There is no…

> That half-adder circuit in there is especially suss. It's really just manipulating 1s and 0s, but -and I've been explicitly told this- no one cares how it actually does it; so long as the truth table matches up.

"so long as the truth table matches up." Yup. Now try getting your chatbot's output to match up.

Your calculator was designed to tell truth. Your chatbot was designed to tell a mash up of whatever its creators managed to scrape from the internet.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#96

Post by jpc0 »

Kim_Bruning wrote: post #92
jpc0 wrote: post #37

Earlier quoted context omitted.

These things are not human, have no agency and cannot be held accountable. We don’t need to restrict them from doing things, we need to default to allowing them to do things. “My agent did XYZ because I allowed it to” is the only valid argument that can be made, and not not every agentic harnass is just a shell toolcall, every one I have built has a specific defined usecase and toolcalls that allows it to execute tha…

So as you scale up, the stakes and the difficulty go up too. Visualize an optimizer on a high dimensional landscape. (The canonical form) ... Ok, I find that hard too. Instead, imagine a river running down to the sea. You put a dam in front of it. It'll pool into a lake and find every crack and crevice. If you didn't survey the land properly or made any error whatsoever, the water will find a way down. (And there's m…

My argument isn’t against those that actually put in the effort and got held accountable, it’s against the “we gave our agent bash and internet and it hacked xyz”.

Bash and internet in that example might be highly abstracted but it’s still bash and internet.

Just look at the replies in this very comment thread, it’s pretty much “We tried nothing and we’re all out of ideas”

In the only other discipline you mentioned, engineering, there would be reviews and any negligence would result in direct action against the engineers that signed off.

For some reason when it comes to building AI harnesses the default response is an ad piece and people shilling how smart and sophisticated the model is.

Imagine a dam collapsing and the engineering firm pumping how smart and tricky water is.

If it’s hard be more diligent, move fast and break things doesn’t really apply in all cases.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#97

Post by chrisjj »

birdsongs wrote: post #79
chrisjj wrote: post #77

Earlier quoted context omitted.

You've been fooled by a next-token predictor.

Does it matter whether it meets the criteria of what you define as "intelligent" when the "next token predictor" throws a backdoor into openssh?

No. But no-one is seriously suggesting intelligence is needed for persistent code fuzzing. Just as no-one is suggesting its needed for computerised chess.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#98

Post by chrisjj »

infinite_spin wrote: post #85
chrisjj wrote: post #77

Earlier quoted context omitted.

You've been fooled by a next-token predictor.

And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems. If you want to downplay that as nothing more than a fancy auto-complete, be my guest, I lose nothing from that.

> And yet this "next-token predictor" is able to churn out well tested, valuable solutions, to complex problems.

Same for countless computer programs from Excel to Google web search. Intelligence has nothing to do with it.

Throw an unimaginable amount of computer power at a problem, and there will always be people who cannot imagine the results to be anything but the creations of intelligence.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#99

Post by throwawayqqq11 »

ben_w wrote: post #82
mcv wrote: post #60

Earlier quoted context omitted.

Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware? Should weapon manufacturers test their weapons by starting wars? I would expect more responsibility from a government agency.

> Is it AISI's job to waste the time and resources of open source projects by attempting to spread malware? Before LLMs got good enough to do this, lots of people were dismissive of their capabilities and didn't take seriously the idea that this was a risk to protect against. Then again, before LLMs, people were saying that obviously nobody would be dumb enough to put an AI on the internet where it could hack anyone,…

Apparently, they (agencies and big-ai) are not performing smoke tests before running capability tests. All the recent headlines of rogue agents shouldnt exist.

Re: How a Texas student blew the whistle on a rogue AI hacking attempt

#100

Post by vel0city »

CoastalCoder wrote: post #63
blagie wrote: post #55

Earlier quoted context omitted.

Personally, I think gun companies should be liable for any harm done by their products as well. We want rule-of-law, and in the US, people should have an absolute right to bare arms, as in the second amendment. Free market forces can then determine appropriate prices, insurance, and protective measures to make sure those guns are managed safely. If I want an F35 and an Abrams, that's okay, so long as Lockheed and Gen…

A bit of a tangent, but I never understood the legal reasoning for how (states having the right of well-regulated militias) implies (individuals having the right for private ownership of arms).

Its not just implying the right is for the people, it's directly stated. It's "the right of the people to keep and bear arms". It doesn't say "the right of the militias" to keep and bear arms".
Post reply on HN