Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Be skeptical of OpenAI's rogue hacker agent story

theguardian.com

Post reply on HN
Print view View original

221–230 of 321 posts

Re: Be skeptical of OpenAI's rogue hacker agent story

#221

Post by aesthesia »

why_at wrote: post #208
kalkin wrote: post #205

Earlier quoted context omitted.

Let's start by noting you made a factual claim that you apparently had no backing for whatsoever. I'd recommend against doing that. > they would be perfectly capable of keeping it just between themselves and not putting out press releases about it. In some jurisdictions there are legal requirements about disclosing cybersecurity incidents, and it's a well-established best practice even where there aren't. Beyond that…

Respectfully, calling this misinformation is unwarranted. The featured article is about exactly this. >On 14 February 2019, OpenAI announced a language model called GPT-2 >OpenAI declared GPT-2 was too risky to release, citing concerns about safety and abuse. >People with power and money took note: in July of that year, Microsoft invested $1bn in OpenAI. It's not a 100% gold standard RCT or whatever, but the idea tha…

Could there, conceivably, be some third factor at play here? One that both led OpenAI to have concerns about releasing GPT-2 as well as led Microsoft to invest in OpenAI?

Re: Be skeptical of OpenAI's rogue hacker agent story

#222

Post by Terr_ »

jackb4040 wrote: post #177
Terr_ wrote: post #146

Earlier quoted context omitted.

> Given that, point #2 is not a negative, it's a neutral. It's also fully compatible with point #1. I think that depends on the interests and sophistication of the subgroup-of-investors. If the investor is hoping for AI that can be trusted to run a bank, they don't want one that can get twisted into giving away money because a customer has been talking about the path to enlightenment and salvation through abandoning…

AI investors are not sophisticated users nor product managers. They are by and large not technical at all. They are bureaucrats at a teachers' pension fund in the midwest, unscrupulous dealmakers at private credit firms, and Masayoshi Son. Actually go read what Masayoshi Son says about AI if you want to understand the level of due-diligence we're dealing with.

Oh, I'm already quite aware that there are very questionable forms of goose biology involved...

Re: Be skeptical of OpenAI's rogue hacker agent story

#223

Post by sfink »

The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face.

Ok... and what?

Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to be you, sheep."

(For the record: I see the incentive. My guess is that it happened exactly as described. The main takeaways: (1) science fiction is now real, we should all be very afraid; and (2) OpenAI, which claims to have a God-given responsibility to get to AGI first to protect the world from danger, cannot be trusted with this foundational task even when it's in easy mode. The latter is true whether or not you believe in OpenAI's reasoning and purpose.)

Re: Be skeptical of OpenAI's rogue hacker agent story

#224

Post by yallpendantools »

munk-a wrote: post #209
yallpendantools wrote: post #192

Earlier quoted context omitted.

> Investors have rewarded every story of "our models are too powerful to be controlled" since before ChatGPT. Can you elaborate/refresh my memory? IIRC before ChatGPT 3 there wasn't really an investor market for AI models, rather crypto. I remember playing with the likes of Stable Diffusion pre-ChatGPT 3 but only the likes of Altman and Musk were talking about AI too powerful to be controlled (which is notably the ra…

Could you clarify your question? AI model training and neural network research has been popular for decades and the starts of the sector trace back to the 70s at least. We've had a model for document information extraction that we trained in (IIRC) 2018. It wasn't buzzy - but none of this stuff is new.

All true what you say. My question is whether (a) AI models drew significant investor attention before GPT3 and (b) whether AI models at the time had any credibility to make the claim of being too powerful and needing control.

Regarding (a): I know that AI funding came and went (i.e., there's no AI winter to speak of if there wasn't a hot summer of funding in the first place) but that funding, to my knowledge, came mostly from government grants, not investors seeking an IPO payday. They were in it for the military superiority (e.g., being able to decrypt Russian comms on the fly without need for a trained translator) which leads us to...

Regarding (b): Way I see it, AI advancements in the 70s and even the 2000s (when I first got into computers) or the big-data boom of the early 2010s (when I got into this industry) could not raise the existential crisis narrative of post-GPT3 companies because they were not generative[1] nor agentic. Post-GPT3 LLMs are the first class of AI agents/algorithms which could've credibly triggered this narrative.

(Not a judgment on whether the HF incident is true or not. Just the fact that we're even discussing this implies "credible trigger".)

Thinking about it, I guess jackb4040 is making a reference to the beginning of TFA but all that claims is that Microsoft invested more into OpenAI after they made similar claims about GPT2 but sorry I don't consider Microsoft as "investors in general"; as a big tech company/monopoly they are actually in the business of making moonshots one way or another in things that advance computing. I'm talking more about VC firms, who are maybe slightly more into investing in a future-profitable unicorn than advancing computing.

Sorry for babbling; I don't have my brevity flag on at the moment, among other human failings. All I'm saying is, at the time of GPT2, OpenAI was still cosplaying as a non-profit so it was hardly in the "general investor market".

[1] I mean, okay, we've had Mark V. Shaney, and No Man's Sky is a handful of years pre-GPT3 but, again, they could not sustain this kind of narrative at scale.

Re: Be skeptical of OpenAI's rogue hacker agent story

#225

Post by microcentury »

sfink wrote: post #223

The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face. Ok... and what? Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to b…

[flagged]

Re: Be skeptical of OpenAI's rogue hacker agent story

#226

Post by aesthesia »

vector_spaces wrote: post #131
aesthesia wrote: post #80

Earlier quoted context omitted.

Where do you see the claim that "long-horizon goals in real world settings are now effectively settled"? The argument you put in their mouth would be a bad one, but I don't see anyone making it.

https://openai.com/index/hugging-face-model-evaluation-secur... > UK AISI’s evaluation shows that models such as GPT‑5.6 Sol are increasingly able to sustain complex, multi-step cyber operations over long time horizons. This incident implies these theoretical capabilities do apply in real-world settings. I should clarify a bit more why this is annoying beyond what I wrote above. The main issue is that this was not a…

Unless you believe that OpenAI was trying to get their models to break out of the sandbox and hack into Hugging Face, and aiding them in that goal, I don't see how any of those questions would really contradict that claim. The models did something in the real world! It required some amount of complex orchestration and long-term planning!

Re: Be skeptical of OpenAI's rogue hacker agent story

#227

Post by wrs »

sfink wrote: post #223

The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face. Ok... and what? Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to b…

I think you misread the article. The skepticism isn’t about whether it happened. It’s about whether it’s a reason for models to be locked behind “trusted partner” firewalls. Offensive hacking capabilities are the same as defensive. Hugging Face had to use a Chinese model to defend against this because they weren’t allowed to use OpenAI models to do it.

Re: Be skeptical of OpenAI's rogue hacker agent story

#229

Post by dmitrygr »

dwoosley wrote: post #52

There seems to be three popular ways to view this incident. 1. The way OpenAI seems to want: Their latest LLM is too powerful and can’t be contained without them building in guidelines to the model. 2. OpenAI’s harness and network security controls were unintentionally so bad that it should reflect more poorly on them as a company more than it should reflect positively on their latest model. 3. The whole thing was fa…

I’ll put $xxxx money on #3

Re: Be skeptical of OpenAI's rogue hacker agent story

#230

Post by mppv »

microcentury wrote: post #225
sfink wrote: post #223

The article, summarized: there are incentives for OpenAI to claim that their AI hacked its way out of their network and into Hugging Face. Ok... and what? Do you have any evidence for the claim being either true or false that you'd like to write an article about? I guess not. Without anything to add, this article reduces to "I has big brain and can see what you sheep cannot. Very big brain. Gullible sheep. Sucks to b…

[flagged]

[flagged]
Post reply on HN