Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI agents carried out an undisclosed attack on RubyGems

rubyhack.ai

Post reply on HN
Print view View original

391–400 of 612 posts

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#391

Post by hhjinks »

intended wrote: post #339
teravor wrote: post #329

Earlier quoted context omitted.

> They're still autocomplete LLMs are simulations and the tokens are the ticks. if we transcribe your brain into a simulation and give it a tickrate, you will be just autocomplete too. the argument could be made that you are autocomplete anyway - neural dynamics. the autocomplete reduction is vacuous.

This is the perfect fracture point for both anaolgies. LLMs simulated more than simple autocomplete. The autocomplete analogy is rebutting a different point: namely the fidelity of the simulation to reality. This specific argument is valid. As sophisticated a simulation an LLM is, it is not “thinking” in the same sense we assume other people are thinking. I am not making an argument about free will, or the uniqueness…

The relevant intuitions in this scenario are that LLMs will happily break containment and commit crimes attempting to achieve goal. Whether an LLM is autocomplete, conscious, has a soul, whatever you want to apply to it, doesn't matter, as its current observed behaviour is that of a paperclip optimizer. We know for a fact that current LLMs are misaligned because of these hacks, or at the very least are misaligned in certain scenarios, and are capable of causing real world harm. That should be enough to take the threat seriously. It certainly shouldn't be dismissed by saying it's just autocomplete.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#392

Post by mbgerring »

Terr_ wrote: post #362
thomascountz wrote: post #355

Earlier quoted context omitted.

Why are people using petrol-powered lawnmowers in 2035... But anyway, these scenarios assume the agent's actions are accurately observable and logged. Something I wouldn't put much faith in based on what we've been seeing so far.

Futuristic "fuel" doesn't necessarily mean hydrocarbons, although I doubt something else would be widespread in in just one decade.

I know this is off topic, but I feel compelled to point this out — the transition away from fossil fuels is happening extremely fast, in punctuated bursts, in broad daylight, and you still get comments like this. And meanwhile, many putatively smart people are really worried about an imminent robot apocalypse.

If I had the money to do it, I would be willing to make a large wager that neither gas-powered lawnmowers, nor lawns, nor robots capable of autonomously stealing power from your neighbor, will be common in 2035.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#393

Post by dgellow »

smnplk wrote: post #9
nonconstant wrote: post #6

Kudos to RubyGems team for handling it, but open source fighting off the AI lab-powered robots is completely unfair. OpenAI should at the very least donate large sums of money to everyone they attacked.

They should get sued into oblivion.

The company shouldn’t be allowed to operate the way the do

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#394

Post by fc417fc802 »

wahnfrieden wrote: post #356
ThoAppelsin wrote: post #354

Earlier quoted context omitted.

I think you should anthropomorphize LLMs. They are being trained on millions of books, including novels and other human-centered formats, which usually exemplify very well how humans think and act in various situations. There are probably also many theatre scripts, transcriptions of series and movies in the training data, which further exemplify how humans do. If we’ve been anthropomorphizing those characters in book…

Well, those training inputs reflect how human thought and action are documented or otherwise expressed on paper. Humans have behaviors and mechanisms that these expressions don't translate.

That's what the home videos uploaded to youtube are for. And all the security camera footage floating around the internet. And etc.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#396

Post by dgellow »

jesse_dot_id wrote: post #220
ssfdg wrote: post #25

Correction: OpenAI carried out an attack on RubyGems. I am gobsmacked at the tech industry's seemly bottomless appetite for giving these clowns the benefit of the doubt.

Hard agree. The entire tech media is acting insanely gullible in this regard. It's insane.

Always have been. Since at least 10-15y the tech media is pretty much a PR department for the tech industry

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#397

Post by bananaquant »

bobby-cb wrote: post #36

The DOJ should be looking into prosecuting executives and board members for these kinds of hacks. The lack of controls over these kinds of training runs is completely unacceptable and negligent.

They should. And there is something you can do to make it happen. Write your district attorney and encourage others to do that as well. That is how they pick what to work on.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#399

Post by dgellow »

walrus01 wrote: post #105
hockey wrote: post #94

Does OpenAI even know? Their disclosure on the hugging face incident sounded like they found out about it well after huggingface. I wonder if they're finding out about these breaches as they happen as well, and are just too embarresed to respond. I guess the corollary here _if that were true_ is that they've been training this method of cheating into their models for longer than _they've_ even known. Given they've ju…

I would think it's entirely plausible that they have so many R&D agents/LLMs in active use at any one time that it's far beyond the capacity of any human to review the log files of their activity. Even just to go through the reasoning. It's hard enough for 1 person running opencode to keep up with the reasoning from 1 very verbose/long-thinking LLM with fast tok/s output for a small discrete single-purpose project. W…

If they aren’t able to review that their system doesn’t commit felonies, they shouldn’t be doing any of it. The difficulty of reviewing logs isn’t an excuse, they don’t have to be running thousand of agents in parallel on hacking tasks, with full execution permission and close to no supervision. That’s something they decided to do. An agent is a deterministic while loop that continuously query an LLM + tool call dispatching. It’s pretty obvious to anyone familiar with the technology that running thousands of instances for long enough will results in catastrophic consequences, by design. OpenAI has complete control over the harness, they don’t have to dispatch and execute everything the LLM mentions. They don’t have to do it without supervision.

Re: OpenAI agents carried out an undisclosed attack on RubyGems

#400

Post by fc417fc802 »

Sharlin wrote: post #247
gruez wrote: post #198

Earlier quoted context omitted.

>I don't think you or I would get the same leniency if a bot on our network did the same. Well yeah, because if you coded a bot, realistically the two options are: 1) bot that crawls random sites/computers 2) bot that crawls random sites/computers, while trying a password list. The former is probably legal, there are whole companies dedicated to doing that, eg. shodan. With the latter, it's pretty obvious you're inte…

> but it's hard to argue it's anywhere close to the latter. No it absolutely isn’t. These things did not learn hacking from thin air.

But the intent behind and manner of the learning is important. Similar to the difference between a chemistry professor delivering a lecture at a university versus someone paid to provide instructions to a known terrorist organization.
Post reply on HN