Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

121–130 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#121

Post by zombot »

mnicky wrote: post #86

I think points that deserve more attention in the current public discourse are: - This should be a huge wakeup call for everybody. - We are lucky that it wasn't a case of an agent running a virology lab benchmark that decides to hack a lab and tries to synthesize something. - It also shows apparent lack of competence and oversight from OpenAI: how is it that they didn't quickly find that agent is breaking the sandbox…

Move fast and break things! Chances are that nobody will ever be held accountable and when push comes to shove, the tax payer will bail you out.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#122

Post by gilbetron »

simonw wrote: post #49
killjoywashere wrote: post #45

Where does this leave formal verification? Are we just shit-out-of-luck at this point? You can formally verify everything about an airplane's code, but if any of that is wrong, ChatGPT might decide that the best way to help you win the Nobel Prize is to take down the airplane your chief rival for the prize is currently in.

I think formal verification has never looked better. The main reason formal verification has never really taken off is that it's difficult. LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers. I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it…

The first reason formal verification has never taken off is that it's difficult. The second reason, that most people don't get to because of the first reason, is that formal verification is really brittle. It is only verified under the very specific setup of the problem. Close doesn't count in math.

The first reason prevents humans from engaging with them, the second reason is what will make it difficult even for LLMs. I mean, I'm glad we are trying, but I'm dubious they will be the panacea some people proclaim.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#124

Post by mirashii »

simonw wrote: post #77
mirashii wrote: post #76

Earlier quoted context omitted.

> I expect OpenAI would agree with you that "the lack of a real sandbox or airgap is nothing short of irresponsible on OpenAI's part". They have clearly invested a lot in those systems for their production models, but in this case they had deliberately turned a bunch of them off for a research project. I'm not sure I can reach the same conclusion. There's no reason that stronger network protections would need to be t…

> There's no reason that stronger network protections would need to be turned off for a research model that's not supposed to have internet access. They didn't turn off the network protections - it looks like they used the same system as other sandboxed agents, which is to block all network access except for an HTTP proxy which only allows read traffic to PyPI and a few other package installation targets. I'm hoping…

> They didn't turn off the network protections

Right, but this is my point in saying I can't reach the same conclusion that they've invested a lot into these systems for production models. Either they had better network protections, and turned them off in their "sandboxed testing environment" (their words), or they don't have more comprehensive network protections at all and rely on this one, extremely thin layer even in production.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#126

Post by js8 »

gilbetron wrote: post #122
simonw wrote: post #49

Earlier quoted context omitted.

I think formal verification has never looked better. The main reason formal verification has never really taken off is that it's difficult. LLMs are significantly more familiar with Lean and Rocq and TLA+ than most software engineers. I think the cost of trying to build systems that adopt formal verification may have just dropped low enough that companies will consider them when previously the ROI didn't look like it…

The first reason formal verification has never taken off is that it's difficult. The second reason, that most people don't get to because of the first reason, is that formal verification is really brittle. It is only verified under the very specific setup of the problem. Close doesn't count in math. The first reason prevents humans from engaging with them, the second reason is what will make it difficult even for LLM…

Close can count in math - fuzzy logic and probability are a thing.

But I think you have it backwards. Close doesn't count in IT security. "Almost secure" means unsecure. Security is the compelling argument for formal verification.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#127

Post by ktimespi »

mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

Agreed, deterministically bounding permissions is the way. I don't know how this is not the first approach that people take.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#128

Post by jackb4040 »

mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

I agree with every word of this except "irresponsible". We don't have enough information to say anything for certain. But based on their incentives and track record of similar behavior, the burden of proof lies with OpenAI to prove they didn't prompt the thing to achieve this exact outcome. The most likely scenario is that they were purposefully executing their responsibility to their shareholders to produce their own Mythos moment.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#129

Post by cvoss »

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure.

Governments should immediately begin leveraging this technology on the defense side (literally defense, not euphemistically "defense") to harden critical infrastructure. Turn the prompts around and use it to identify and correct weaknesses.

Governments should also take very seriously their now moral obligation to treat this technology not just as "a powerful thing that might be abused" but as an actual weapon of war in need of international regulation analogous to nuclear arms. Fast but careful and forward-thinking work in legislation and treaties needs to be a top priority for all major governments.

Post reply on HN