Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

261–270 of 440 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#261

Post by bonoboTP »

efficax wrote: post #121
Ancv123 wrote: post #104

Earlier quoted context omitted.

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt). So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

It's very possible but somewhat slower. PyTorch and CUDA have flags for determinism. It won't work across all different GPU models though, but it will get you bitwise equal results on the same GPU.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#262

Post by shepherdjerred »

gizajob wrote: post #198
KingOfCoders wrote: post #187

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking the unsecure service, no tripwire, researchers find the hack days/weeks/months later, fix it, but don't secure the sandbox and the service was hacked a second time, again without being monitored by security researchers. Then security researchers create a black…

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either. Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared. OpenAI h…

OpenAI has a pretty clear definition of AGI

> OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work

https://openai.com/charter/

Re: Timeline of the OpenAI accidental attack against Hugging Face

#263

Post by simonw »

la64710 wrote: post #258
Wowfunhappy wrote: post #242

Earlier quoted context omitted.

"Complete subservience and complete intelligence do not go together." I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want. Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty w…

The whole thing seems to depend upon AI agents objective ie to achieve some objective by any means possible and ignoring any guardrails. The article did not clarify if openAI had any guardrails to begin with while conducting this experiment. For all the talks around how much they invest in AI safety one would expect them to have these common sense guardrails in place or is it just a case of some school children letti…

OpenAI didn't have any guardrails in place - they were training a model at a point much earlier than when guardrails start being implemented.

The guardrail was meant to be that the agents were running in a locked-down environment with no internet access. The entire problem came about because it turned out that sandbox didn't hold.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#264

Post by flatline »

simonw wrote: post #255
flatline wrote: post #245

Earlier quoted context omitted.

I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn. https://www.secondtalent.com/resources/ai-trends-in-china/

Oh, it's the OpenRouter number: https://finance.yahoo.com/technology/ai/articles/china-ai-mo... Those numbers aren't credible IMO because OpenRouter only see traffic for people who have chosen to route their traffic through OpenRouter. If you do that, you're much more likely to be experimenting with alternative models. They have no insight at all into people who point their applications directly at OpenAI or Anthropi…

I agree about OpenRouter. The AI Gateway number [0] is likely the figure that was actually coming to mind. Moreover, Qwen models alone have overtaken the previously-dominant Llama models in hf downloads by quite a margin.

Real question, and a refinement to my previous statement: would you find it more surprising if over 25% of worldwide inference was running on Chinese open-weight models, or not? I personally would not be shocked.

[0] https://vercel.com/blog/ai-gateway-production-index-july-202...

Re: Timeline of the OpenAI accidental attack against Hugging Face

#265

Post by simonw »

shepherdjerred wrote: post #262
gizajob wrote: post #198

Earlier quoted context omitted.

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either. Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared. OpenAI h…

OpenAI has a pretty clear definition of AGI > OpenAI’s mission is to ensure that artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work https://openai.com/charter/

There's also the private definition reportedly agreed between Microsoft and OpenAI, leaked in December 2024: https://techcrunch.com/2024/12/26/microsoft-and-openai-have-...

> The two companies reportedly signed an agreement last year stating OpenAI has only achieved AGI when it develops AI systems that can generate at least $100 billion in profits.

That was necessary because of the deal they had from a while ago where Microsoft would lose access to OpenAI's technology once OpenAI achieved AGI.

Apparently they renegotiated that away in April 2026: https://openai.com/index/next-phase-of-microsoft-partnership...

> Revenue share payments from OpenAI to Microsoft continue through 2030, independent of OpenAI’s technology progress, at the same percentage but subject to a total cap.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#266

Post by simonw »

flatline wrote: post #264
simonw wrote: post #255

Earlier quoted context omitted.

Oh, it's the OpenRouter number: https://finance.yahoo.com/technology/ai/articles/china-ai-mo... Those numbers aren't credible IMO because OpenRouter only see traffic for people who have chosen to route their traffic through OpenRouter. If you do that, you're much more likely to be experimenting with alternative models. They have no insight at all into people who point their applications directly at OpenAI or Anthropi…

I agree about OpenRouter. The AI Gateway number [0] is likely the figure that was actually coming to mind. Moreover, Qwen models alone have overtaken the previously-dominant Llama models in hf downloads by quite a margin. Real question, and a refinement to my previous statement: would you find it more surprising if over 25% of worldwide inference was running on Chinese open-weight models, or not? I personally would n…

I wouldn't be too surprised by that, given both the size of the Chinese market and the enormous price discount you get compared to the US models.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#267

Post by talon8635 »

etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

How fast the goal posts shift.

Of course it’s exceptional agent capability when compared to all of history previous to one week ago.

Like, I know everyone here obsesses over AI and uses and follows it very closely, but come on guys. Yes, it is wild that these things are this good. This technology is still brand new. It could t do basic maths a year ago.

Sure, the OAI team was negligent in various ways, and they should be held culpable. But that doesn’t detract from the true black magic that is these modern models.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#269

Post by emp17344 »

uncivilized wrote: post #249
androiddrew wrote: post #169

I wish we could stop sensationalizing this about the AI and really just understand the incompetence of the labs disabling an internet connection in a sandbox.

Hacker News doesn’t have the wherewithal to understand that this is just marketing by OpenAI.

The whole site is suffering from AI psychosis.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#270

Post by mvdtnz »

pmarreck wrote: post #148
RGS1811 wrote: post #46

Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

> Complete subservience and complete intelligence do not go together.

Isn't this contradicted by the centuries of slavery in our history? Or is the author arguing that the people who were enslaved did not have human-level intelligence (which would be rather a problematic claim)?

Post reply on HN