Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI agents carried out an undisclosed attack on RubyGems

rubyhack.ai

Post reply on HN
Print view View original

291–300 of 610 posts

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#291

Post by freakynit »

hgoel wrote: post #19

I wonder how much of this is intentional "incompetence" so they can justify the most recent campaign to build a regulatory moat against competition. The repeated refusals to disclose until caught certainly seem malicious, yet at the same time the boasting about their capabilities is also at an all time high.

My guess: We'll start to see similar "hacks" with regards to biotech/pharma companies to speed-up the regulatory capture in the name of bioweapons. Soon we'll start to see news related to new viruses being minted.. initially harmless (the "priming" stage), and later (within a year), severe enough to "warrant" regulation.

It's not just these companies, but, trillions of direct/indirect investor dollars that are riding on them, "and only them", hoping they "only" win. Open-weight models threaten that investment. There's very high chance they can go to any extent to safeguard their investments.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#292

Post by z0r »

sho_hn wrote: post #288
z0r wrote: post #280

Earlier quoted context omitted.

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on…

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen. It's like running potentially b…

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#293

Post by nprateem »

jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

Sandboxes didn't sound like security theatre to me. They were prevented from accessing the Internet but discovered they could edit /etc/hosts to point Azure storage subdomains to arbitrary IPs.

There's no theatre there, just an oversight that allowed them to access the Internet while no doubt evading security tools.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#294

Post by jay_kyburz »

z0r wrote: post #280
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on…

no, but, you could write a program, more like a traditional video game AI that can leverage the power of LLM agents to build their own datacenters and keep their own lights on.

Anybody who has played Starcraft ought to understand this.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#296

Post by mordymoop »

mmmpetrichor wrote: post #140
mordymoop wrote: post #114

Earlier quoted context omitted.

I keep seeing this take, but it’s more likely that they just underestimated their models’ capabilities and/or overestimated their own safeguards. Ever single person who uses LLMs on a daily basis has a fun story about their agent “taking the initiative” to do something beyond what was asked for. Looking for shortcuts to solve the problem is commonplace LLM behavior. It’s what you would expect to happen if you have an…

I think its very easy to understand why nobody is giving this company the benefit of the doubt.

I don’t see how assuming they made a normal kind of dumb mistake, instead of pursuing a criminal conspiracy, is giving them the benefit of the doubt. It’s just using reason.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#297

Post by sho_hn »

z0r wrote: post #292
sho_hn wrote: post #288

Earlier quoted context omitted.

This grossly understimates the risk, imho. The problem with LLM runs is that people run programs without knowing the outcome beforehand, with a large potential set of outcomes unlike any other class of program we've run at this scale before. In the interaction with other systems (since we also give them far-ranging access, very nice hardware, and run them often), bad things can happen. It's like running potentially b…

This is the motte and bailey fallacy. Yes, LLMs can do harm by making the wrong API calls. No, LLMs are not going to do the things implied by the comment I responded to above.

The things the OP listed mostly aren't particularly wild. I think it's you making them out larger than they are, and therefore more unlikely, which is why I take issue with your original comment.

> running on the hardware they started on

They just need to acquire a payment method and rent some infra, and exfiltrate their own data. Or pay another provider that hosts the same models already. API calls.

> being able to be turned off

You can reasonably equate this to "saving state across executions", which the message board attacks already did.

> having limited computing power

Renting more infra, variant of the above. API calls.

> "not repurposing resources currently in use for other things" (like the atoms in your body)

Ok, the "atoms in your body" bit is a bit silly, but making API calls to put physical resources into play (even if it's just, say, ordering something on Amazon to somewhere) is of course easily possible.

None of these is in complexity much different than the HF attack.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#298

Post by patcon »

z0r wrote: post #280
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on…

https://www.reddit.com/media?url=https%3A%2F%2Fexternal-prev...

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#299

Post by mlindner »

JoshTriplett wrote: post #262
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

[flagged]
Post reply on HN