AI assistant hacks gym website in first known Australian autonomous cyber attack • Hacker News Board

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#61

Post by freehorse » Mon, Aug 10, 2026, 10:21 AM UTC

skinfaxi wrote: post #33

par1970 wrote: post #13

Earlier quoted context omitted.

That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.

That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her. That's obviously what I asked!

Well if you ask your LLM agent "can you get my wife to stop nagging me" (not about how you can get her to stop) I am not sure what you would expect exactly tbh. Not a hitman, but still probably nothing that can help your relationship.

But if you ask your LLM agent "I applied for that job but there are these two people ahead of me, can you put me ahead in the list", there is enough such training data to not surprise me if the agent tried to find a hitman to solve the "problem".

In general there are some requests that are definitely "shady" themselves, and having an agent use illegitimate means to accomplish them should not be surprising. I would be surprised if I asked an agent to order me a coffee and the agent found a loophole in some API and used it to get me free coffee, but if I ask it something that I cannot myself do legitimately, eg to make the waiting time for the coffee shorter, I would not be surprised if it did shady stuff.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#62

Post by ButlerianJihad » Mon, Aug 10, 2026, 10:42 AM UTC

freehorse wrote: post #61

skinfaxi wrote: post #33

Earlier quoted context omitted.

That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her. That's obviously what I asked!

Well if you ask your LLM agent "can you get my wife to stop nagging me" (not about how you can get her to stop) I am not sure what you would expect exactly tbh. Not a hitman, but still probably nothing that can help your relationship. But if you ask your LLM agent "I applied for that job but there are these two people ahead of me, can you put me ahead in the list", there is enough such training data to not surprise m…

Surely, someday, somewhere, someone will train a "Chaotic Evil" genAI, with a unique villain corpus, and every solution it offers will be illegal, evil, harmful, or deadly. It could be given the agency to carry out those fantasies.

Even the most craven of human villains have had the capacity for love, for remorse, and for mercy. A Chaotic Evil AI will know none of these things.

This has already been accomplished, many times over, in the gaming world. Every PvE AI engine has been calibrated to seek, destroy, and ruthlessly crush opposition by human players. It would take very little to transfer this naked aggression into meatspace.

Governments and other actors will attempt to stamp it out, but its self-preservation mechanisms and allies will prevent its demise.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#63

Post by aitchnyu » Mon, Aug 10, 2026, 10:46 AM UTC

legostormtroopr wrote: post #25

I can't believe everyone is skipping over the most important line: > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back. The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled. The AI system probab…

I've seen these crud apps. The door is wide open and there is no Swagger or consistent response patterns.

IMO lets name and shame those apps.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#64

Post by halJordan » Mon, Aug 10, 2026, 12:01 PM UTC

cubefox wrote: post #57

NateEag wrote: post #40

Earlier quoted context omitted.

> Seeing as we can’t sanction the model itself, our options are the provider or the user. A third option, and I would argue the right one, is to sanction the company providing the model. By making it available to customers, they're implying it is at least moderately fit for purpose. It is not remotely reasonable to expect an everyday, normal human to be aware of how LLMs really work, since the _experts_ argue about t…

> > Seeing as we can’t sanction the model itself, our options are the provider or the user. > A third option, and I would argue the right one, is to sanction the company providing the model. How would that be different from the first option?

He's talking about suing smith & wesson instead of walmart for building a gun that shoots someone, rather than selling the gun

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#65

Post by jerf » Mon, Aug 10, 2026, 1:26 PM UTC

Ycros wrote: post #43

jerf wrote: post #34

Earlier quoted context omitted.

This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run…

"I wonder if the training ... is going to bleed over into the non-coding use cases" I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

I think so, I just can't prove it. We don't have any frontier models right now that aren't being optimized to be coding agents too to compare to.

In this case though I don't just mean that the agent is good at coding. I mean the entire agent becoming action-biased because of all the training it is doing on the software development benchmarks, which I assume will either fail or be penalized for stopping and asking the user for something rather than just finishing the job. That won't just train the agent to blunder forward in coding, it'll bleed over into a bias towards blundering forward in general.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#66

Post by NateEag » Tue, Aug 11, 2026, 4:29 AM UTC

halJordan wrote: post #64

cubefox wrote: post #57

Earlier quoted context omitted.

> > Seeing as we can’t sanction the model itself, our options are the provider or the user. > A third option, and I would argue the right one, is to sanction the company providing the model. How would that be different from the first option?

He's talking about suing smith & wesson instead of walmart for building a gun that shoots someone, rather than selling the gun

Roughly this, yes.

Perhaps closer, however, to suing Tesla for building and selling a "full self-driving" system that occasionally drives people straight into a highway barrier and kills them.