Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

271–280 of 319 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#271

Post by matsemann »

thadk wrote: post #107

Simon's retelling is more compact but it also invites anthropomorphization of the sharing of the familiarity with the message board which re-emerged a few times. Zvi's retelling handles this better. Zvi speculates that the secret message board familiarity was carried because it had been trained into the May-and-subsequent models: https://thezvi.substack.com/p/openai-trained-its-models-for-...

Simon's really doesn't bring anything useful to the table.

One question I'm stuck with after reading is why. Why did the agents do these things? I get them being adamant on getting internet, but why did they continue? Why hack HuggingFace?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#273

Post by mofeien »

gizajob wrote: post #198
KingOfCoders wrote: post #187

Security researchers expose an unsecure service to agents who were instructed to hack software and called that a sandbox. Agents escape the sandbox by hacking the unsecure service, no tripwire, researchers find the hack days/weeks/months later, fix it, but don't secure the sandbox and the service was hacked a second time, again without being monitored by security researchers. Then security researchers create a black…

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either. Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared. OpenAI h…

I don't think air gapping will work: even human security researchers recovered a 378-bit key from a Samsung Galaxy S8 through a power LED of a speaker two devices away.

And accessing memory in a specific sequence can generate radio signals that can be picked up by a mobile phone at a distance: https://arxiv.org/html/2409.02292v1

Re: Timeline of the OpenAI accidental attack against Hugging Face

#275

Post by fantasizr »

yread wrote: post #257
doc_ick wrote: post #201

Earlier quoted context omitted.

It’d be more like “car accidents occur, so let’s add seat belts, air bags, etc…”.

... and speed limits

licensure (age and competency), an entire insurance industry, domestic and international regulations - and so forth

Re: Timeline of the OpenAI accidental attack against Hugging Face

#276

Post by prohobo »

bonoboTP wrote: post #261
efficax wrote: post #121

Earlier quoted context omitted.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

It's very possible but somewhat slower. PyTorch and CUDA have flags for determinism. It won't work across all different GPU models though, but it will get you bitwise equal results on the same GPU.

Both of your comments are illuminating :p

So, we could technically debug a prompt's output? I get that there are too many steps to actually step thru, but what if there were checkpoints? At least you could isolate behaviors to specific sections of a neural network?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#277

Post by Sleaker »

mvdtnz wrote: post #270
pmarreck wrote: post #148

Earlier quoted context omitted.

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

> Complete subservience and complete intelligence do not go together. Isn't this contradicted by the centuries of slavery in our history? Or is the author arguing that the people who were enslaved did not have human-level intelligence (which would be rather a problematic claim)?

Neither, the author is pointing out the desire of the enslaver, not the actual outcome. But I don't think their logic takes into account access to means to 'outsmart' the enslaver. It's trying to frame it as a single instance equation, not a societal one to try and show the underlying contradiction of desire.

At least, that's what I'm pulling from the quote, have not read the full context.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#278

Post by famouswaffles »

mvdtnz wrote: post #270
pmarreck wrote: post #148

Earlier quoted context omitted.

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

> Complete subservience and complete intelligence do not go together. Isn't this contradicted by the centuries of slavery in our history? Or is the author arguing that the people who were enslaved did not have human-level intelligence (which would be rather a problematic claim)?

Is that complete subservience ? Slave history has tended towards slaves no longer being slaves over long enough time horizons, and not simply because the slave masters were just feeling extra nice. Slaves don't really like being slaves.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#279

Post by mofeien »

springtimesun wrote: post #152

What’s missing to me in all this is: did it succeed in its initial task? And then, did it stop? I feel like whether I should be scared or not hangs on those questions

From TFA: It did succeed in the "accidentally impossible" task, but not at all in the way the problem-setters intended, and rather... at all costs?!

And it wouldn't really matter whether it stopped afterwards, I think. At sufficient model capability a single task set badly enough would end catastrophically upon the agents succeeding at it, no?

Re: Timeline of the OpenAI accidental attack against Hugging Face

#280

Post by deadbunny »

stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

I don't think the problem is that they are training the models to perform cyber attacks, they're training them to be better at coding and problem solving which has the byproduct of them being very capable cyber attack weapons.

Their objective is to solve the problem and they'll use anything they can to solve it.

Anecdotally I was debugging a css issue and opus 4.7 was churning away as I was half paying attention only to see it opening plain css as hex, when questioned wtf it was doing it proclaimed it was verifying 2 files were identical. Thing that make sense to these models wouldn't even cross a greybeard's mind.

Post reply on HN