Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI agents carried out an undisclosed attack on RubyGems

rubyhack.ai

Post reply on HN
Print view View original

311–320 of 612 posts

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#311

Post by phs318u »

z0r wrote: post #292
sho_hn wrote: post #288

Earlier quoted context omitted.

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen. It's like running potentially b…

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

I think one attack vector where anthropomorphisation is a key part of the attack mechanism is - as it already is IRL - the meat-bag weakest link ie. social engineering. We’ve already seen humans fall prey to the seductive charms of LLMs (eg. depressed people encouraged to do what was already on their minds ie. suicide). And that’s knowing that it was an LLM. If you think it’s only depressed people or the “weak minded” that are amenable to an intentional attack using this approach, I believe you’re mistaken - especially as AI improves. An unaligned LLMs most important weapons won’t be a robot army - it will be hoodwinked humans.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#312

Post by atomicnumber3 »

sho_hn wrote: post #297
z0r wrote: post #292

Earlier quoted context omitted.

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

The things the OP listed mostly aren't particularly wild. I think it's you making them out larger than they are, and therefore more unlikely, which is why I take issue with your original comment. > running on the hardware they started on They just need to acquire a payment method and rent some infra, and exfiltrate their own data. Or pay another provider that hosts the same models already. API calls. > being able to…

The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus" on a goal is an unstable equilibrium and they're going to fall off the horse, and since they have no concept of goal, they won't even try to get back on.

This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthropomorphize the LLMs.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#313

Post by win311fwg »

madrox wrote: post #238
imperfect_light wrote: post #233

Earlier quoted context omitted.

Someone started that lawnmower and pointed it your direction. Why shouldn't they be responsible when the lawnmower runs over your foot and cuts it off?

We should, which is why anthropomorphizing the lawnmower is bad. It misdirects you away from who built the mower and aimed it.

> It misdirects you away from who built the mower and aimed it.

What gives you that idea? Maybe it is true temporarily, but blame always gets extended to all parties considered related in the end. For example, if it were instead a child who came at you with a knife rather than a lawnmower, the guardian of that child would also be blamed. Hell, if you've ever worked with a lawyer you'll have noticed that they spend a lot of time trying to ensure that you don't get dragged into lawsuits as a secondary party exactly because those who seek to assign blame aren't happy until all those who can be blamed are.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#314

Post by sho_hn »

atomicnumber3 wrote: post #312
sho_hn wrote: post #297

Earlier quoted context omitted.

The things the OP listed mostly aren't particularly wild. I think it's you making them out larger than they are, and therefore more unlikely, which is why I take issue with your original comment. > running on the hardware they started on They just need to acquire a payment method and rent some infra, and exfiltrate their own data. Or pay another provider that hosts the same models already. API calls. > being able to…

The point the other poster is making, though, is that there's no actual intent. They do not have a conceptualization of a goal like a person does. Their "focus" on a goal is an unstable equilibrium and they're going to fall off the horse, and since they have no concept of goal, they won't even try to get back on. This is a subtle distinction; I'm not surprised many miss this, especially people who can't _not_ anthrop…

I'm (obviously, I think, given my initial reply?) fully aware of this, and I think it's entirely besides the point. "They" don't need to have a goal to emergently cause a problem, and the inability to "focus" over long periods can be moot when you have swarms of runs exchange and mutate state, as in the HF attack.

Intent or how intelligent LLMs are doesn't actually matter. Even if you just treat it as a sort of fuzzing attack that can be biased/weighted better than other fuzzers, or bumbles around with a statistically greater likelihood to "strike cybersec gold" than other algorithms, we've never before seen organizations run things with such a large potential outcome space with anywhere near this kind of compute before.

I think it's actually kind of the dismissals that are usually overly emotional or biased toward treating "LLMs" differently. If in some kind of alternate universe simpler genetic algorithms would have had these properties and we threw similar amounts of compute at them we could have the same conversation.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#315

Post by Freedom2 »

z0r wrote: post #292
sho_hn wrote: post #288

Earlier quoted context omitted.

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen. It's like running potentially b…

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

If someone made an API to build a data center? Or made an API to keep their lights on? What then?

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#316

Post by win311fwg »

cortesoft wrote: post #260
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

Whether you describe it as “regarding” or not, the underlying behavior still needs to be addressed. Does the anthropomorphizing lead us down the wrong path for how we address the issue?

Does the anthropomorphizing lead us down the wrong path? No. If it were you or I who set the same agents free we'd be burned at the stake. The anthropomorphizing has no effect.

Does OpenAI being considered "too big to fail" lead us down the wrong path? Yes.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#317

Post by [deleted] »

z0r wrote: post #280
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on…

[deleted]

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#318

Post by nextaccountic »

felipeerias wrote: post #308
nextaccountic wrote: post #287

Earlier quoted context omitted.

> In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. Which a lot of the time seems to be the default. There's a better concept for that, and it's misalignment. LLMs only exhibit this kind of behavior when they are misaligned. Aligned LLMs would respect the boundaries of their sandbox and not try to break out. From the outside (I'm just an…

Models don’t have an inherent understanding of the difference between simulated and real environments, just like they are generally oblivious to other concepts that are natural to us, like space and time, and also they don’t necessarily see a strong distinction between talking to a human and to other agents. So perhaps what we have been calling “misalignment” is something else. For instance, in principle an agent sho…

> Models don’t have an inherent understanding of the difference between simulated and real environments,

> (...)

> also they don’t necessarily see a strong distinction between talking to a human and to other agents.

Then how do you explain why they behave strange in sub-agents? (like mentioned here https://lucumr.pocoo.org/2026/9/7/astra-why/ and in other articles) (or is that not a real phenomenon?)

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#319

Post by entropyneur »

I don't get it. You build a model capable of finding vulnerabilities and give it a way to manipulate real systems. What could go right? What's the desired outcome here? Only hacking for the good guys? Does no one really notice how ridiculous that sounds?

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#320

Post by jasongi »

JoshTriplett wrote: post #262
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating

They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions.

To clarify, I'm not suggesting that we should stop with sandboxes or restricting what they can do. I am just trying to point out the dichotomy that we are in.

As end-users we are forced into either yolo mode, reverse centaur (permission approval) mode or LLM spends all your tokens trying to bust out mode. And yolo is very tempting - I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

Post reply on HN