Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s accidental attack against Hugging Face is science fiction that happened

simonwillison.net

Post reply on HN
Print view View original

291–300 of 475 posts

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#291

Post by mirashii »

didibus wrote: post #266
mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

Fair on the terminology angle, that said in this case, they had proper guardrails no? They were running in a sandbox, but it was able to find an exploit out of the guardrails.

Clearly not. Good security is about defense in depth. The service that was exploited was meant to be an offline, frozen package cache, it shouldn't have been anywhere near a network boundary to even allow this through. A single exploit in one service should not have been able to bypass all network filtering, monitoring, and alerting.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#292

Post by mirashii »

ashleyn wrote: post #205
mirashii wrote: post #70

> Given the absence of guardrails there was nothing to prevent the model from attempting to break out of that sandbox, break into Hugging Face, and read the answers from there instead. I've said this many times before and I'll continue to shout it, but using the term "guardrails" to refer to anything that's either (a) in-context, or (b) a probabilistic classifier (including using other LLMs), is an irresponsible abus…

What we call "guardrails" in an AI agent, we would refer to as "honor system" in human actors. Or, in a more direct sense, the AI should be set up in an environment such that no matter how hard it may try to call $PART_OF_EXPLOIT_CHAIN, the environment just isn't capable of it (ideal) or doesn't permit it to do it.

I like "honor system" as a term. I've been looking for the right term to replace the irresponsible usage of guardrails with, and best I've had so far is the pinky promise protocol.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#293

Post by [deleted] »

cvoss wrote: post #129

The technology held by private AI companies is warfare-capable technology. Imagine the prompt: "Use all available resources to disable the power grid of ." The resource cost that prevents scaling up such a war machine is, what, just the cost of building data centers and its ongoing power bill? Cheap and easy compared to nuclear infrastructure. Governments should immediately begin leveraging this technology on the def…

[deleted]

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#294

Post by npiano »

blovescoffee wrote: post #284
seydor wrote: post #234

Science fiction usually includes intent, and that the AI has inherently evil motives and it has a goal. LLMs are zombies, and the fact they do evil things means they were either trained to be too aggressive in their drunkenness or that problem-solving leads inevitably to evilness. But science fiction also refers to deprogramming evil robots.

> LLMs are zombies How do you know?

Because there is no internal feedback loop. There is no internal process gathering information from the external world and using it to modify its future behaviour. There is only the beginning of the calculation and then the end.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#295

Post by docjay »

1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym.

2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to note mistakes it makes, how the plan to solve the problem takes shape, etc. They keep the whole thing headless with no output, then just UDP a prompt into it and leave for the weekend? No they don’t, but if they do then that answers a lot about their complete disconnect from how their models work.

3. Language models are a two player game; text in, text out. What prompt was given to a sub-agent that resulted in it immediately attempting to exit the sandbox (which it apparently knew it was operating within) and continuing in a feedback loop of ‘function call -> result’ until it hacked the Gibson? “Analyze and summarize the ” simply does not result in ‘hmm…this sounds like a benchmark question, I bet Hugging Face has the answer in a database. ’

…there are more, but a lot of the story kind of stinks.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#296

Post by npiano »

IshKebab wrote: post #220
BiraIgnacio wrote: post #215

Call me obtuse but the way this is being portrayed by the companies involved, media, seems a little odd to me. Didn't the model + harness do what was asked? If I ask a coding agent to write a very clever piece of code and it turns out impressively clever, it did what I asked.

> Didn't the model + harness do what was asked? Depends exactly what they asked it to do, but it very clearly didn't do what was intended, or what an honest human would do. Stop trying to find a gotcha.

It very clearly didn't do what they said they told it to do. We do not know what was truly intended.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#297

Post by weare138 »

There will inevitably be some people who dismiss this story as a dishonest marketing trick by OpenAI to make their models sound terrifyingly effective. I found 81 instances of the term “marketing” in the Hacker News discussion of the incident.

But a marketing stunt isn't the only possibility. Until this is all verified by investigators or 3rd party experts we can't rule out an act of corporate espionage either. Extraordinary claims require extraordinary evidence.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#298

Post by estearum »

docjay wrote: post #295

1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym. 2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to…

1. It's been extremely well established for multiple generations of models that they have no problem detecting when they're being evaluated. Pretty sure it was Opus 4.8 that the independent evaluators literally filed an assessment that said "We have no assessment to make as [model] consistently detected it was being evaluated, making our assessments untrustworthy."

2. Regardless of whether the model was being watched closely during this evaluation, do you actually think a sensible safety guard is "have humans watching it 24/7?" What does "watching it" even mean? Watching network logs? Uhh for your entire company? At all times? After you just deployed a system whose entire purpose is to "do a shitload of work way faster?"

3. You're asking "why was this system that was designed to behave agentically behave agenitcally?" Again: that's the whole point. It was designed that way because it's more valuable than having a repeated turn-based interaction. Thus also it becomes more dangerous.

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#299

Post by meander_water »

docjay wrote: post #295

1. Wouldn’t the model need to know it’s answering benchmark questions, as well as the name of the benchmark, in order for the idea of finding the answer in a database somewhere to even surface? The whole point of benchmarks is to present the question or problem as a standard prompt, not explain that it’s a test called ExploitGym. 2. Nobody was watching it? I don’t mean “babysit the dangerous autocomplete”, I mean to…

1. The benchmark is run with a python script - https://github.com/sunblaze-ucb/exploitgym using an agent harness. I suspect they used codex. So the model has access to the environment and could trivially inspect its own source code, which has lots of references to exploit gym and docs relating to it.

2. Hanlons Razor

Re: OpenAI’s accidental attack against Hugging Face is science fiction that happened

#300

Post by furyofantares »

simonw wrote: post #236
manux wrote: post #232

Earlier quoted context omitted.

Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is…

I don't think this exposes an alignment failure, because the test here was run with the alignment features deliberately turned off. OpenAI said: > We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. It was a test of raw capabilities of the underlying model.

Is turning the alignment features off more like multiplying "don't be evil" by 0, or by -1?
Post reply on HN