Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI agents carried out an undisclosed attack on RubyGems

rubyhack.ai

Post reply on HN
Print view View original

331–340 of 612 posts

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#331

Post by JoshTriplett »

jasongi wrote: post #320
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions. To clarify, I'm not suggesting that we shoul…

> I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

So you would approve of breaking into HuggingFace and RubyGems?

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#332

Post by chii »

jasongi wrote: post #320
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions. To clarify, I'm not suggesting that we shoul…

> They're still autocomplete

it's like saying our brain is just some chemical chain reactions. True, but also irrelevant.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#333

Post by lelanthran »

JoshTriplett wrote: post #262
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating.

Autocomplete in a feedback loop is still autocomplete, no?

Doesn't the process look like this:

    (context + prompt + "reason about this")
                       |
                       V
                Reasoning Output
                       |
                       V
    (everything + Reasoning Output + "Now do final output")
                       |
                       V
         (Final output seen by prompter)
???

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#334

Post by jasongi »

InsideOutSanta wrote: post #302
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

> In my experience Do you work for one of these companies? If not, you have no experience with any of the models that carried out these attacks, and your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models. Also, the lawnmower analogy is a worse way of understanding LLMs than anthropomorphising th…

> Lawnmowers never break out of your garden and into your neighbor's house and eat their dog because you've told them to be careful when mowing the lawn because the neighbor's dog pooped in it.

All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.

Your example is still anthropomorphising - LLMs don't seek revenge. They complete the prompts they are given. If your task is not achievable without sandbox escapes, or you throw unnecessary amounts of compute at open-ended tasks like preparing for a future quiz then you shouldn't be surprised that the preparation eventually turns to cheating and hacking.

> your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models.

I don't but I don't think there's anything wrong with discussing how we can already observe publicly available models work around sandboxes and permissions and make the connection that maybe this is what that behaviour looks like when a more capable model exhibits it.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#335

Post by frabcus »

z0r wrote: post #280
JoshTriplett wrote: post #262

Earlier quoted context omitted.

> Why would autocomplete know If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating. > In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. LLMs need to stay carefully contained, and if they're ever breaking the guardrails put around them, they're misaligned and should not be…

You should unplug, my friend. These words are fantasies. LLMs are token prediction engines and they aren't going to build their own data centers. They can't keep their own lights on. The real world is full of fractal details that a disembodied token prediction engine will never come to grips with. Even if they started to, you could probably defeat them with the kind of logic used to combat evil sentient computers on…

The question isn't just about LLMs.

The labs have the specific goal of automating ML engineering, and with the code automation they have are getting close. They are competing to brute force maths, presumably as that is similar long horizon and skillset to persistently brute force making new/better ML training algorithms.

They will then run those, and they won't be LLMs any more. What we think about token predictions isn't relevant if the architecture allows continual learning of recurrent networks.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#336

Post by wahern »

dghlsakjg wrote: post #218
wahern wrote: post #205

Earlier quoted context omitted.

A strict liability crime is something of an oxymoron. Crimes always require intent, the mens rea element. The question is intent for what. If somebody drugged you without your knowledge and you were charged with a DUI, you would have a defense--no intent to become intoxicated. The strict liability means once you choose to become intoxicated, you're liable for driving intoxicated, even if in some other context your in…

Are you a lawyer? Fairly certain that the entire point of strict liability is that mens rea is not required for certain crimes. As in, if I meant to travel at 70 and was instead doing 100 it doesn’t matter that I sincerely meant not to speed and did not know I was speeding, I can still be convicted even if the judge believes I had no intent.

See my response at https://news.ycombinator.com/item?id=49668714

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#337

Post by frabcus »

cortesoft wrote: post #260
jasongi wrote: post #204

> The agents clearly regarded what they were doing as hacking. To butcher the quote about Oracle: Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doi…

Whether you describe it as “regarding” or not, the underlying behavior still needs to be addressed. Does the anthropomorphizing lead us down the wrong path for how we address the issue?

Sort of...

It's how we anthropomorphise corporations which leads us down the wrong path. OpenAI is no longer fully aligned with humanity.

Somehow we call corporations "people" sometimes when it makes them more powerful, but suddenly stop anthropomorphising and don't call them "evil hackers, misusing computers", when they both make and let loose an irresponsible hacking AI.

It's bizarre. Of course, just like AI, corporations are neither people nor machines. They're a dynamic, agentic, persistent other.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#338

Post by BHSPitMonkey »

qarl wrote: post #231
mcphage wrote: post #230

Earlier quoted context omitted.

Oh, you can, no problem. But you’re also wrong , and getting downvoted because of it.

I'm wrong for pointing out that this entire line of thought - AI companies are trying to shirk blame by pinning it on their software - is unhinged? No friend, I am not. It is hysterics pure and simple.

A few excerpts from https://news.ycombinator.com/newsguidelines.html worth a careful re-read:

> In Comments

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#339

Post by intended »

teravor wrote: post #329
jasongi wrote: post #320

Earlier quoted context omitted.

> If you still believe LLMs are "autocomplete", your cache of understanding about them needs invalidating and regenerating They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions. To clarify, I'm not suggesting that we shoul…

> They're still autocomplete LLMs are simulations and the tokens are the ticks. if we transcribe your brain into a simulation and give it a tickrate, you will be just autocomplete too. the argument could be made that you are autocomplete anyway - neural dynamics. the autocomplete reduction is vacuous.

This is the perfect fracture point for both anaolgies.

LLMs simulated more than simple autocomplete.

The autocomplete analogy is rebutting a different point: namely the fidelity of the simulation to reality.

This specific argument is valid. As sophisticated a simulation an LLM is, it is not “thinking” in the same sense we assume other people are thinking.

I am not making an argument about free will, or the uniqueness of human thought, just that the correspondence to how humans reach conclusions and how the simulation produces outputs do not match on a 1:1 basis; as a result attributing traits builds incorrect intuitions.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#340

Post by ascorbic »

pdonis wrote: post #46
qarl wrote: post #43

Earlier quoted context omitted.

I think it's more likely they want to call attention to the fact it was the result of agents, rather than shift blame. I'm pretty sure everyone knows that OpenAI is liable for the software they create and run.

> I'm pretty sure everyone knows that OpenAI is liable for the software they create and run. Are they? What legal consequences have they suffered?

If any of the orgs that were hacked decided to sue OpenAI they'd probably have a good chance of winning. It's just that none of them has.
Post reply on HN