Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

211–220 of 308 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#211

Post by famouswaffles »

simoncion wrote: post #206
famouswaffles wrote: post #189

Earlier quoted context omitted.

>reliably instruct a dumb-as-bricks CPU Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Like, you're not making any sense here. None of the things that make this possible with CPUs is remotely relevant here, and the fact that you don't seem to understand this but act so smug is strange.

> Yeah...a "dumb as bricks CPU", which is obviously something frontier llms are demonstrably not. Just as the immense amount of scaffolding around the dumb-as-bricks CPU enables extremely sophisticated and useful things to be done with that pile of fused sand and copper, the immense amount of scaffolding around the dumb-as-bricks LLM enables very sophisticated and useful things to be done with that pile of linear alg…

LLMs are not the "stupid bit in the middle." They're almost the entire value. LLMs were wildly useful before any sort of scaffolding. They are not "dumb as bricks". They are highly capable, flexible, intelligent prediction machines.

The only one confused here is you, and you've still not managed to tell us in an actionable way how exactly CPU scaffolding is relevant here. Tell us, if it's so easy, or make your millions selling it. We're all waiting.

I'll give you a hint. CPUs never had to interpret the meaning of arbitrary content in order to do their job, and LLMs do.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#213

Post by wolttam »

androiddrew wrote: post #169

I wish we could stop sensationalizing this about the AI and really just understand the incompetence of the labs disabling an internet connection in a sandbox.

As written it sounds like you're saying that it was incompetent of the labs to disable the sandbox internet access?

They tried to disable open internet access but the models zero-day'd their Artifactory package registry and got internet access anyway.

No sensation... that's just what happened.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#215

Post by nightshift1 »

simonw wrote: post #93

I think one of the most interesting details here might be tucked away in that first bulletin point: > May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.) The…

The slide at 14:06 say:

By june 11: Highly persistent experimental, internal-only model begins training.

I am not sure what that means. Are they preserving notes/memories and context between runs?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#216

Post by flatline »

KingOfCoders wrote: post #187

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking the unsecure service, no tripwire, researchers find the hack days/weeks/months later, fix it, but don't secure the sandbox and the service was hacked a second time, again without being monitored by security researchers. Then security researchers create a black…

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.”

This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#217

Post by famouswaffles »

simoncion wrote: post #192
tsimionescu wrote: post #142

Earlier quoted context omitted.

> "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?". I feel this completely misunderstands the problem, and the vast gulf between an LLM and a CPU. First and most importantly, the set of behaviors of a CPU is extremely constrained, and we have a very simple model for which behaviors are safe and which are not. Writing to addresses between X and Y, executing certain in…

> Secondly, CPUs can absolutely be used to hack other people. This is more correctly phrased as "Every general-purpose computer can be run any arbitrary program, assuming it has the storage required to load that program.". Despite that fact, we've managed to learn how to write programs that run on those computers that fail to give attackers who have control of the inputs to those programs control of the instructions…

[dead]

Re: Timeline of the OpenAI accidental attack against Hugging Face

#218

Post by hoten »

nightshift1 wrote: post #215
simonw wrote: post #93

I think one of the most interesting details here might be tucked away in that first bulletin point: > May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.) The…

The slide at 14:06 say: By june 11: Highly persistent experimental, internal-only model begins training. I am not sure what that means. Are they preserving notes/memories and context between runs?

That's how I interpreted it, but now I'm wondering if they mean "this model gives up far less often"..

Re: Timeline of the OpenAI accidental attack against Hugging Face

#219

Post by bamboozled »

swader999 wrote: post #17

This is clearly out of control, Zero parent supervision.

It’s insanely incompetent. What’s more wild is the present at Blackhat with “full transparency” almost boasting about how powerful their models are. Basically just endlessly doing and allowing foolish things to happen to lead to a law breaking outcome.

Not to take away from the technology which is wild in itself. But there was literally zero oversight into what was going on at OpenAI. Whether that was intentional, it’s hard to say …

Re: Timeline of the OpenAI accidental attack against Hugging Face

#220

Post by hoten »

KingOfCoders wrote: post #210
gizajob wrote: post #203

Earlier quoted context omitted.

it just escaped your sandbox.

How can it escape an "install package " proxy? reducePrivs() serve get(package) { secPackage = secure(package) getBinaryFromArtifactory(secPackage) } I would think the code is very small and easier to verify, it doesn't especially have the ability to write files and act as a message board as Artifactory did. And even if the agent tries to hack that, the attack surface is 1000x smaller and the possibility also much sm…

I mean, it's just the same problem. The machine still has Internet access. It doesn't need to.

The entire package manager repository could just be in an offline cache. They don't need Internet to give their agents access to tons of software.

Post reply on HN