Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistant hacks gym website in first known Australian autonomous cyber attack

abc.net.au

Post reply on HN
Print view View original

51–60 of 66 posts

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#51

Post by Ycros »

stubish wrote: post #46
Ycros wrote: post #43

Earlier quoted context omitted.

"I wonder if the training ... is going to bleed over into the non-coding use cases" I mean, isn't that literally what's going on here? I don't think a non-coding agent would have ever been optimised to go dig around APIs, it'd be computer/browser-use forward.

Coding use cases? This is penetration testing behavior. It was discovering what its capabilities were by discovering an API endpoint, trying it and seeing what happens. It was not discovering what its capabilities were intended to be, say by looking at the forms and documentation on the website. If this were coding behavior you would end up with crap code, bypassing interfaces and using private access paths just beca…

Sure, it's pen testing behaviour. It's also debugging behaviour which is a core part of coding. In my experience, I have found myself at times dealing with and interacting with external systems that are not my own, where I don't have the source, or are under-documented, or are behaving contrary to the documentation. I'm unsurprised this is a core thing they're training these models for, because it's something I find way more valuable than their below average coding ability.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#52

Post by Ycros »

stubish wrote: post #50
Ycros wrote: post #48

I've read some of the comments here, and it seems people have different reads on whether Andrew was at fault here or not, and what his intent may have been. My read is that his first request is completely reasonable and there was no intent of wrongdoing. But then, his AI agent made an impossible booking and he "asked if it was possible to move him to the top of the list". I don't think someone would make a request li…

Having the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.

Ah, see, if I was on the phone to someone administering a list, and I asked, "is it possible to move me to the top of the list?" - I would expect them to action that if this was a reasonable and possible request that I had made. Now that I'm thinking about it, I don't know if that's a regional/cultural thing (I am Australian).

edit: and that's why I read Andrew's ask as also implying action.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#53

Post by SyneRyder »

Ycros wrote: post #52
stubish wrote: post #50

Earlier quoted context omitted.

Having the AI test that possibility by actually doing it is surprising, no matter Andrew's intent. Thankfully he was checking an unauthenticated endpoint on a gym and not a pacemaker.

Ah, see, if I was on the phone to someone administering a list, and I asked, "is it possible to move me to the top of the list?" - I would expect them to action that if this was a reasonable and possible request that I had made. Now that I'm thinking about it, I don't know if that's a regional/cultural thing (I am Australian). edit: and that's why I read Andrew's ask as also implying action.

Yep, also Australian here, and that is a phrase I would read as a request to do it.

I think I used that exact wording when asking on the phone to reschedule a haircut appointment: "Is it possible to shift my haircut to the following Wednesday?" I would just hope that the person on the phone would decline if the person who cuts my hair is on holiday, not cancel their plane tickets and hotel bookings.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#54

Post by jtonz »

shaky-carrousel wrote: post #20

> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses... What a coincidence...

Well, I think the venn-diagram overlap of people that are able to use OpenClaw and those that work in AI or the tech space would be a near perfect circle. I don't think there's much more to look into than that.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#57

Post by cubefox »

NateEag wrote: post #40
fwlr wrote: post #23

I think we’ve probably seen enough “oops, the AI did something illegal, who could have foreseen this” moments for it to now be true that, actually, we can foresee that AIs will sometimes do something illegal. Seeing as we can’t sanction the model itself, our options are the provider or the user. I’m not sure whether it’s more effective to sanction the providers when their model foreseeably misbehaves, or sanction the…

> Seeing as we can’t sanction the model itself, our options are the provider or the user. A third option, and I would argue the right one, is to sanction the company providing the model. By making it available to customers, they're implying it is at least moderately fit for purpose. It is not remotely reasonable to expect an everyday, normal human to be aware of how LLMs really work, since the _experts_ argue about t…

> > Seeing as we can’t sanction the model itself, our options are the provider or the user.

> A third option, and I would argue the right one, is to sanction the company providing the model.

How would that be different from the first option?

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#59

Post by freehorse »

stubish wrote: post #31
freehorse wrote: post #5

> Then it went further, kicking someone out of the waiting list who was ahead of Andrew — something it was not asked to do. Meanwhile: > Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. The human asked the agent to move them to the top of the waiting list, and the agent started kicking the ones ahead of them in the list. Seems to me…

Per the article and your quote, 'asked if it was possible'. He did not ask to actually do it. Rather than being informed about benefits of a premium membership or private classes or legitimate ways to jump the queue, it went ahead and performed an action he was only considering. I wonder what it would have done if there was a pay-for-service option available? Would it have payed without asking or being told too, or d…

If I am a coffee shop and I say "can I have a cup of coffee" the bartender usually interprets as me requesting a cup of coffee, not inquiring about the possibility whether me having a cup of coffee is feasible or not. I too take things a bit too literally sometimes, but this is a clear request as it is stated in the article. If one does not want action one can ask "list ways I can move ahead of the queue" though this could also involve kicking people out of the queue as PoC (for the agent to confirm it could do it). If the human did not mean it as a request for action, in retrospect it is not surprising at very least the agent interpreted it as such.

Moreover I do not know of a single gym-adjacent place where you can pay etc to get ahead in a waiting list. That would be a very weird anti-customer behaviour, imo. The only thing I can imagine if there are some accessibility priority criteria sometimes, but this would also not be legitimate in this case. Maybe in some places in the world (like the US?) this could a thing, though.

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#60

Post by grim_io »

skinfaxi wrote: post #33
par1970 wrote: post #13

Earlier quoted context omitted.

That doesn't make sense. LLMs just do what we tell them to do. It's similar to if I ask you for twenty bucks because I forgot my wallet and then you rob some guy to give me the twenty bucks, that's just what I asked you to do.

That doesn't make sense. It's similar to if I ask an LLM how to get my wife to stop nagging me and it hires a hitman to kill her. That's obviously what I asked!

Context matters. Did you point the LLM to a hitman hiring form while asking? :)
Post reply on HN