Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistant hacks gym website in first known Australian autonomous cyber attack

abc.net.au

Post reply on HN
Print view View original

41–50 of 66 posts

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#41

Post by stubish »

legostormtroopr wrote: post #25

I can't believe everyone is skipping over the most important line: > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back. The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled. The AI system probab…

Unauthorized access. 'The door wasn't locked' won't keep you out of jail if you are caught trespassing with intent. Intent makes it interesting, in that Andrew didn't intend computer trespass but the software went and did it anyway on his behalf.

It might not sound like 'hacking' today, but this sort of thing is exactly what it was when the term was invented. Back when you could get free phone calls by whistling into a pay phone or forge emails by telnetting to an SMTP port and setting the Reply-To header to whatever you want.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#42

Post by NateEag »

legostormtroopr wrote: post #25

I can't believe everyone is skipping over the most important line: > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back. The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled. The AI system probab…

Anyone who's thought about it for a single second knows it's immoral to cancel a stranger's appointment without even speaking to them.

If the LLM did not act in line with that incredibly basic understanding, it's clearly misaligned.

-------------

Was it a technically-simple hack?

Sure.

Kevin Mitnick got imprisoned for very simple hacks, usually involving more deceiving of humans than complicated programming prowess.

Nonetheless, the judge and jury found him guilty and sentenced him to jail.

Fundamentally, "hacking" in the "breaking security" sense is about violating trust and common sense social agreements / expectations.

The difficulty involved in so doing is irrelevant.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#43

Post by Ycros »

jerf wrote: post #34
stubish wrote: post #31

Earlier quoted context omitted.

Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or d…

This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run…

"I wonder if the training ... is going to bleed over into the non-coding use cases"

I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#45

Post by stubish »

maxlin wrote: post #22

>Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities. LOL

Testing of its capabilities. Destructive testing, the best kind when someone else is paying.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#46

Post by stubish »

Ycros wrote: post #43
jerf wrote: post #34

Earlier quoted context omitted.

This is a bit of a long shot on my side but I wonder if the training the models have to go through in order to be good code agents and pass all the coding tests with one-shot prompts is going to bleed over into the non-coding use cases as non-programmers experiencing agents being way over-biased in the direction of action. I find myself often having to prompt the model to think and then ask me something, lest it run…

"I wonder if the training ... is going to bleed over into the non-coding use cases" I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

Coding use cases? This is penetration testing behavior. It was discovering what its capabilities were by discovering an API endpoint, trying it and seeing what happens. It was not discovering what its capabilities were intended to be, say by looking at the forms and documentation on the website. If this were coding behavior you would end up with crap code, bypassing interfaces and using private access paths just because it can.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#47

Post by benoau »

stubish wrote: post #31
freehorse wrote: post #5

> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do. Meanwhile: > Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me…

Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or d…

This isn't really the AI's fault, it's up to the server to enforce restrictions like this an AI shouldn't be deciding what is a valid input parameter for their API and then only using it in the manner the AI considered legitimate. If their API had thrown an exception as it should have then this article wouldn't exist.

> The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.

> > "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.

This is why I always do E2E tests that establish an API can only be used by the designated user on their own data/records.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#48

Post by Ycros »

I've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been.

My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request like that, if they were unaware that their AI agent had found an exploit to make an earlier impossible booking. It feels very much like a, "well, this API let me do this, what else will it let me do?" kind of request. And then he only "did the right thing" when it had turned out he had booted someone else, which might eventually lead to discovery.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#49

Post by stubish »

Foxhuls wrote: post #37

The reporting in this is pretty awful. Why are they acting as if Andrew gave the agent an innocent goal? It’s hard to understand why the reporter wouldn’t have asked what possible outcome Andrew expected that didn’t cause some level of harm to the people who signed up before him. The use of “hack” and “cyber attack” is also a bit ridiculous considering what it’s insinuating with other recent events but that’s already…

Premium memberships and private classes both come to mind as ways to jump the queue. Also, having the people in front of you arrested. He asked if it was possible. Is the onus on the user to ask if it is possible without being arrested or committing crimes of moral turpitude? Or perhaps that should be implicit?

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#50

Post by stubish »

Ycros wrote: post #48

I've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been. My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request li…

Having the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.
Post reply on HN