Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

151–160 of 298 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#151

Post by estearum »

moron4hire wrote: post #81
wizzwizz4 wrote: post #68

Earlier quoted context omitted.

But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how…

I'm talking about OpenAI, not GPT 5.x Flash Uranus Edition Brought to You by Costco, specifically because I recognize the model as just a tool. OpenAI was, at the very most generous interpretation, massively incompetent and negligent.

Is someone arguing otherwise?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#153

Post by satvikpendem »

pmarreck wrote: post #148
RGS1811 wrote: post #46

Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

Can you unformat this, it's quite annoying to read on mobile

Re: Timeline of the OpenAI accidental attack against Hugging Face

#154

Post by KingOfCoders »

simonw wrote: post #89
KingOfCoders wrote: post #74

Earlier quoted context omitted.

No because he doesn't ask the right - and to me, subjectively, obvious - questions.

Who am I supposed to be asking questions of here? I was writing about the new things we learned from the Black Hat video. On TikTok this article's hook would be "I watched the Black Hat video so you don't have to".

I think for the power you have and how many people listen to you, you should have added context. All of it is made as if without prompt or direction, agents on their own initiative, over weeks collaborated to hack Hugging Face - which too me, sounds highly doubtful.

You transporting this without any context makes it seem as you agree with the narrative of OpenAI.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#155

Post by KingOfCoders »

gordonhart wrote: post #102
KingOfCoders wrote: post #52

"The agents found a Modal-hosted insecure app with a weak API key, then used that to stage an attack against Hugging Face." Why, what was the prompt? I told Claude today to wire plugins on Linux into a sound pipeline to remove noise. Did some astonishing things, played sound through the pipeline, measured it etc. I told it to optimize my sound for TF2 and it played the spy_decloak samples, measured them and made them…

This was clearly explained by OpenAI in their initial press release on 7/21 [0]: > This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test s…

It does not explain how agents months later would "collaborate" to hack Hugging Face.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#156

Post by mmilunic »

efficax wrote: post #121
Ancv123 wrote: post #104

Earlier quoted context omitted.

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt). So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

Interesting paper by Thinking Machines where they solve this issue.

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

TLDR: It’s actually more about kernels changing with batch sizes, and you can solve it by making these kernels not depend on batch sizes. It took their inference time from 26s to 42s.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#157

Post by skydhash »

simonw wrote: post #137
simoncion wrote: post #129

Earlier quoted context omitted.

> I get the impression that every AI lab is desperately trying... Of course. I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.

If you can figure out how to separate instructions from data in LLMs you should ship the first agent system that's guaranteed protected against prompt injection. You'll make millions.

It’s pretty simple. Both the intake and the output of the LLMs are data and they shouldn’t drive an actuator system (their output shouldn’t be instruction). We already have the same structure in organizations where there’s an army of analysts for information gathering and processing and then the executive department tasked with decisions.

We have even observed that the most effective LLM usage is when paired with an expert in charge of the goals. Dark factory and other automated harnesses (specs engineering and what not) seem to be a dead end. The most impactful approach to this date is an interactive conversation as a succession of small and verifiable tasks.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#158

Post by user43928 »

wizzwizz4 wrote: post #68
moron4hire wrote: post #45

Earlier quoted context omitted.

I've patched many security vulnerabilities in projects without ever once needing to break into a competitor's network.

But you're actually capable of thought. These AI systems aren't: as far as they're concerned, they're predicting the next part of an incident write-up narrated in first-person limited perspective, like the children in Ender's Game showing off their skills in the training simulations. The AI system neither knows, nor cares, about any "external reality" behind it all, or about anything beyond the text, heedless of how…

About knowing whether a scenario is fictional, there was an interesting finding in Anthropic's J-Lens research.

When they benchmarked the model to evaluate whether it would try to blackmail someone in a contrived scenario, the J-Lens showed "fake" and "fictional" in the workspace.

And if edited out, the model was more likely to do the blackmailing.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#159

Post by simonw »

KingOfCoders wrote: post #154
simonw wrote: post #89

Earlier quoted context omitted.

Who am I supposed to be asking questions of here? I was writing about the new things we learned from the Black Hat video. On TikTok this article's hook would be "I watched the Black Hat video so you don't have to".

I think for the power you have and how many people listen to you, you should have added context. All of it is made as if without prompt or direction, agents on their own initiative, over weeks collaborated to hack Hugging Face - which too me, sounds highly doubtful. You transporting this without any context makes it seem as you agree with the narrative of OpenAI.

Beyond a whole lot of online conspiracy theories I haven't seen anything that suggests to me that OpenAI aren't not telling the truth about what happened here.

I find the Black Hat presentation in particular very credible. Also the Hugging Face technical report.

(As an example of something I don't find credible: https://openai.com/index/responding-next-frontier-critical-c... is a total nothing burger. It's the other end of the credibility scale from the Black Hat talk.)

Re: Timeline of the OpenAI accidental attack against Hugging Face

#160

Post by dan_q »

moron4hire wrote: post #70
dan_q wrote: post #37

Earlier quoted context omitted.

> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. OpenAI is clearly run by dummies and subpar engineering talent. > The model is obviously impressive Speak for yourself.

Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues. It's just really weird. Why does everyone feel the need t…

> Speaking of that "obviously impressive" line, I'm getting really tired of something like that line seemingly needing to be included by anyone doing any criticism of agentic systems. The most common form of it is "these models are obviously useful" midway through a bunch of arguments about environment, data provenance, skill atrophy, or even correctness issues.

> Why does everyone feel the need to equivocate?

Blame the mods here for that. This is the first week since 2023 I haven't had a HackerNews account banned for directly critiquing LLM code generators.

> I want to tell these people to just cut it out. It's demeaning to their own position.

I couldn't agree more. It's a rhetorical disaster.

Post reply on HN