Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI assistant hacks gym website in first known Australian autonomous cyber attack

abc.net.au

Post reply on HN
Print view View original

21–30 of 66 posts

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#21

Post by bulder »

mingus88 wrote: post #18
par1970 wrote: post #11

Earlier quoted context omitted.

> Seems to me like it was doing what it was asked to do? Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."

“Get me there as fast as possible. Hey! I never said you should speed!” This is literally the bad genie / monkey’s paw plot. Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.

Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#22

Post by maxlin »

>Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list.

The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.

LOL

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#23

Post by fwlr »

I think we’ve probably seen enough “oops, the AI did something illegal, who could have foreseen this” moments for it to now be true that, actually, we can foresee that AIs will sometimes do something illegal.

Seeing as we can’t sanction the model itself, our options are the provider or the user. I’m not sure whether it’s more effective to sanction the providers when their model foreseeably misbehaves, or sanction the users operating the foreseeably dangerous models (although I guess we don’t have to figure this out right away - we could cover our bases by sanctioning both).

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#24

Post by cindyllm »

bulder wrote: post #21
mingus88 wrote: post #18

Earlier quoted context omitted.

“Get me there as fast as possible. Hey! I never said you should speed!” This is literally the bad genie / monkey’s paw plot. Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.

Generally when I read a literal genie story, the message isn't 'well it was reasonable of the genie to do this'. It's more commonly either a morality tale of the person being wrong to ask for whatever it was they asked for, or just a "wouldn't it be fucked up if the genie did that huh"

[dead]

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#25

Post by legostormtroopr »

I can't believe everyone is skipping over the most important line:

> "The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.

The AI systemm didn't hack anything, it lightly touched with a feather duster and the server crumbled.

The AI system probably found swagger documentation of each endpoint, figured that the reservation cancellation API was worth a shot, and then found there was no authentication.

What is the "hack" here?

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#26

Post by par1970 »

mingus88 wrote: post #18
par1970 wrote: post #11

Earlier quoted context omitted.

> Seems to me like it was doing what it was asked to do? Maybe it's what he asked it to do, but it's not what he wanted it to do. Which we know because (a) normal people don't want to break the law to get into a gym class, and (b) "But Andrew was shocked by what happened next." and "Alarmed, Andrew asked the agent to undo this."

“Get me there as fast as possible. Hey! I never said you should speed!” This is literally the bad genie / monkey’s paw plot. Give a powerful entity a goal and act shocked when it gets there in ways that aren’t in your best interest.

Putting aside whether Andrew's shock is appropriate, it seems like we agree that current agents at least occasionally do things that are straightforwardly against the interests of the prompter when given mundane prompts like "Get me into this gym class as soon as possible."

How does this look once agents are superintelligent?

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#27

Post by cubano »

shaky-carrousel wrote: post #20

> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses... What a coincidence...

I'm not sure this is the exact advertisement they would want for their business.

Then again...

Re: AI assistant hacks gym website in first known Australian autonomous cyber attack

#30

Post by stubish »

shaky-carrousel wrote: post #20

> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses... What a coincidence...

Interesting he went with publicizing this. Article demonstrates the products he is selling have unknown and untested legal liabilities under Australian Law. You could even add 'Andrew' to the list of people possibly accountable for selling or recommending unsafe or unsuitable products.
Post reply on HN