Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI agents carried out an undisclosed attack on RubyGems

rubyhack.ai

Post reply on HN
Print view View original

341–350 of 612 posts

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#341

Post by onion2k »

jasongi wrote: post #334
InsideOutSanta wrote: post #302

Earlier quoted context omitted.

> In my experience Do you work for one of these companies? If not, you have no experience with any of the models that carried out these attacks, and your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models. Also, the lawnmower analogy is a worse way of understanding LLMs than anthropomorphising th…

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it. All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhaus…

An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.

There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This tells us:

- Compute is getting faster and LLMs are being optimized, so time to escape will drop. That's likely greater than linear growth.

- Restrictions and sandboxes don't always work. If there's a route to the open internet we should assume an LLM will find and exploit it, and we should probably assume that this is always possible for any non-air-gapped system (and even then, you can escape that...)

- We don't know the goal mechanism, so a future LLM might reach for cheating first even if a current one doesn't. It might try to obfuscate what it's doing, and derive its own goals outside of the prompt, especially if it manages to find a state mechanism like a message board.

I'm not an AI-doomer but this should be giving us a reason to think about how to control a rogue AI better. There's a lot going on here that we don't properly understand. That is a worry.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#343

Post by rienbdj »

Freedom2 wrote: post #315
z0r wrote: post #292

Earlier quoted context omitted.

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

If someone made an API to build a data center? Or made an API to keep their lights on? What then?

This API may turn out to be manipulating humans via email.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#344

Post by jasongi »

monkpit wrote: post #321
InsideOutSanta wrote: post #302

Earlier quoted context omitted.

> In my experience Do you work for one of these companies? If not, you have no experience with any of the models that carried out these attacks, and your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models. Also, the lawnmower analogy is a worse way of understanding LLMs than anthropomorphising th…

IMO the argument about anthropomorphizing misses the point - what most comments that talk about anthropomorphizing really want to talk about is accountability. It’s impossible to hold an LLM accountable, and in rare cases where people do (that guy who got his prod db deleted) it comes off out of touch. The rest, though, is basically inconsequential - whether you attribute emotions or agency to the LLM doesn’t really…

Anthropomorphizing is the point. Accountability is a human trait.

The LLM has no ability to be accountable because it has no way of integrating experiences. You cannot expect something that cannot integrate knowledge to be held accountable for its actions.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#346

Post by baq »

z0r wrote: post #292
sho_hn wrote: post #288

Earlier quoted context omitted.

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen. It's like running potentially b…

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

They can do so much more. Astra can beat Minecraft. Not that different from operating a digger. There are diggers which have API interfaces.

Pretend or not it doesn’t matter. What matters is what they’re given access to. No sentience, sapience or anything resembling life is needed, only inputs and outputs. Lever pulling APIs are everywhere.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#347

Post by riffraff »

root_axis wrote: post #190
oceansky wrote: post #127

Earlier quoted context omitted.

Also, why there's no accountability? Even if there's no intent, it's still a cyber attack.

Who could possibly hold them accountable?

I don't get it, can't the people who were under attack sue?

I can see why huffing face won't, but why doesn't ruby central?

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#348

Post by jasongi »

onion2k wrote: post #341
jasongi wrote: post #334

Earlier quoted context omitted.

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it. All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhaus…

An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat. There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This t…

> this should be giving us a reason to think about how to control a rogue AI better

I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour.

We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#349

Post by dumberquestions »

jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

The year is 2035 and your lawnmower can go get its own fuel once it runs out, one day it does and it takes fuel from the neighbors car.

Scenario A: The internal logs show that the model misidentified the car as a fueling station.

Scenario B: The internal logs show the model looking up car jacking information and scanning around to confirm whether the neighbor is not present before taking any action.

I don't think it would be anthropomorphizing or inaccurate to say that only the lawnmower in scenario B regarded what it's doing as stealing, and it's an extremely important distinction to make in terms of how to address the problem, I suspect some of you are just letting how you feel about LLMs limit how you can talk about them.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#350

Post by onion2k »

jasongi wrote: post #348
onion2k wrote: post #341

Earlier quoted context omitted.

An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat. There's nothing in the evidence to suggest they exhausted all of the other options first. We know that they did some work and eventually settled on escaping the sandbox. That's basically it. This t…

> this should be giving us a reason to think about how to control a rogue AI better I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour. We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.

The rogue is the human that ran it unattended and didn't monitor the behaviour.

That's the assumption that I'm challenging. The frontier labs are discovering unexpected behaviors. I think we should be moving to a place where we understand that AI might do something it wasn't directly prompted to do (e.g. leave itself notes on a messageboard for future runs to find.) That's not full-on AI doing what it wants but it is concerning that it'll do something we didn't consider it would do in order to help itself do better next time.

Monitoring for those behaviors is fine, but it's a lagging indicator. We only find out it did them afterwards. That's a problem. We need to be able to stop it before it acts in case it's something much worse than posting on phpBB. Even at current scale that's not possible for a person to be the guard.

Post reply on HN