Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

121–130 of 289 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#121

Post by efficax »

Ancv123 wrote: post #104
RGS1811 wrote: post #46

Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt). So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the order in which these operations happen can result in subtly different final tensors. To reproduce it deterministically you'd have to also reproduce the exact scheduling of all matrix calculations among all the GPU cores (across different physical gpus!) that it took place on, which afaik is currently impossible.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#122

Post by mr_mitm »

KingOfCoders wrote: post #43

"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages." Yeah, my agents also discover what other agents have done on other machines by accident. Agents - that do totally different things all work on the same aim without the humans telling them to do. Either that is a model that is several generations of Claude Code Opus/Fable 5 (my dai…

> NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.

With all due respect, you also aren't evaluating brand new models that haven't been released.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#124

Post by gliall_err »

Meleagris wrote: post #80
dan_q wrote: post #37

Earlier quoted context omitted.

> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. OpenAI is clearly run by dummies and subpar engineering talent. > The model is obviously impressive Speak for yourself.

I don’t believe for a second that they lack the engineering talent. It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.

Talent is not some fungible measure. I know incredibly smart people who can fail at incredibly basic life skills.

"They wouldn't be that dumb" is a meaningless argument. People you don't know can be as smart as anyone on the planet and still make very dumb choices.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#125

Post by azuanrb »

etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

Both can be true. How often do we hear about hacks that ultimately came down to bad defaults or simple security mistakes? That doesn’t mean any script kiddie could have discovered and exploited them.

These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#126

Post by Bjartr »

efficax wrote: post #121
Ancv123 wrote: post #104

Earlier quoted context omitted.

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt). So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable.

You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#127

Post by vasco »

ToValueFunfetti wrote: post #108
dgellow wrote: post #99

Earlier quoted context omitted.

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…

The companies are begging to be regulated for this reason and have been doing so for years. HN's response is generally that this is performative for marketing or seeking regulatory capture or haha anthropic you get what you ask for. Maybe the cynics are right, but there's really nothing inconsistent about the naive view here, once you factor in race dynamics and obligations to investors.

> The companies are begging to be regulated for this reason and have been doing so for years

Regulations are rules that you force on a market, but the actors in the market should not be assumed to be all operating against the regulations before they come into play. Said in other words, these companies don't need to wait for regulation to not destroy the world, if that's truly what they think will happen.

> inb4 someone else will do it

Re: Timeline of the OpenAI accidental attack against Hugging Face

#128

Post by ToValueFunfetti »

moron4hire wrote: post #72
frde_me wrote: post #65

Earlier quoted context omitted.

Knowing how to break into someone else's network will make you a lot better at making your own network secure.

Having experience breaking into networks is not the same thing as learning about the techniques used and the classes of vulnerabilities exploited by attackers.

As a guy who presumably has a lot less experience in security than you, I feel rude even bringing it up: surely you're aware of red teaming? This isn't a novel technique invented for AI- IBM has a page about it, it's what all the best DEF CON talks are about, it's the opening scene of Sneakers, it's the point of CtF games.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#129

Post by simoncion »

simonw wrote: post #118
simoncion wrote: post #113

Earlier quoted context omitted.

> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ens…

I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.

> I get the impression that every AI lab is desperately trying...

Of course.

I wonder how we managed way back in the day to produce systems that can handle untrusted inputs and reliably instruct a dumb-as-bricks CPU what to do based on those inputs. Must have been black magic lost to the mists of time.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#130

Post by efficax »

Bjartr wrote: post #126
efficax wrote: post #121

Earlier quoted context omitted.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

That's not inherent, that's a consequence of performance optimizations. It's absolutely a choice to run those matrix calculations in a way that fails to have predictable execution ordering. It's just that the speed benefits to allowing that are considerable. You can make it trivially deterministic by running single threaded on a cpu, but it's becomes too slow for practical applications if you do that.

well sure, but i mean realistically speaking, we cannot step debug an llm's output to find out what happened given the way we currently execute inference
Post reply on HN