Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

Post reply on HN
Print view View original

171–180 of 260 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#171

Post by layer8 »

satvikpendem wrote: post #153
pmarreck wrote: post #148

Earlier quoted context omitted.

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

Can you unformat this, it's quite annoying to read on mobile

It’s fine in landscape for me, but here you go:

“The problem, and it is a moral problem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, however, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks. However, we also wish him to be subservient. Complete subservience and complete intelligence do not go together. How often in ancient times the clever Greek philosopher slave of a less intelligent Roman slaveholder must have dominated the actions of his master rather than obeyed his wishes! Similarly, if the machines become more and more efficient and operate at a higher and higher psychological level, the catastrophe foreseen by Butler of the dominance of the machine comes nearer and nearer.”

I used https://www.textfixer.com/tools/remove-line-breaks.php.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#172

Post by owebmaster »

uh_uh wrote: post #21
stingraycharles wrote: post #14

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

There are trade-offs here: Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder. Give up too late -> collateral damage while completing the task A.K.A. misalignment.

Asking for the user input isn't giving up

Re: Timeline of the OpenAI accidental attack against Hugging Face

#173

Post by solarengineer »

simonw wrote: post #93

I think one of the most interesting details here might be tucked away in that first bulletin point: > May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.) The…

"This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process."

I am a fan of Asimov and the three laws of Robotics. Itlooks like in OpenAI's world, the three Laws of Robotics would be added later if they were to develop the positronic brain. It may also explain how US Robotics from Asimov's books would have been able to design Robots that only partially adhered to the 3 laws (e.g. the robots in iRobot - the book - which were programmed to allow a human to come to harm through inaction so that the humans could complete their work on the plains of Mercury).

Re: Timeline of the OpenAI accidental attack against Hugging Face

#174

Post by FeepingCreature »

androiddrew wrote: post #169

I wish we could stop sensationalizing this about the AI and really just understand the incompetence of the labs disabling an internet connection in a sandbox.

As AIs become more capable, the level of competence required to avoid disaster likewise goes up over time.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#175

Post by paytonjjones »

efficax wrote: post #121
Ancv123 wrote: post #104

Earlier quoted context omitted.

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt). So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

llms are not strictly deterministic in the sense that even if you had the RNG state, context, and prompt you would likely not get an identical output even if there was no other randomness involved, because the concurrent scheduling of the massive amounts of floating point calculations can produce different results, since floating point arithmetic is not truly associative [(a+b)+c can differ from a+(b+c)] and the orde…

That's very interesting, I wonder if this applies also to models quantized to ints like (-1,0,1), and I wonder if the labs could maintain frontier performance if they removed floating points but arbitrarily scaled up the parameters.

Edit: the Thinking Machines article in the other comment gets into this a bit

Re: Timeline of the OpenAI accidental attack against Hugging Face

#176

Post by FeepingCreature »

KingOfCoders wrote: post #43

"More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages." Yeah, my agents also discover what other agents have done on other machines by accident. Agents - that do totally different things all work on the same aim without the humans telling them to do. Either that is a model that is several generations of Claude Code Opus/Fable 5 (my dai…

The agents you get to use are the agents that "behaved well".

Re: Timeline of the OpenAI accidental attack against Hugging Face

#178

Post by InsideOutSanta »

etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

In a functioning system, I would say that there would have to be some kind of government oversight over companies training models of this intelligence, and that OpenAI should be prevented from continuing their work until they get their act together.

But I guess in the actual world we live in, this is just something that happens, and we all shrug and move on and hope that nothing worse is going to happen tomorrow.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#179

Post by FeepingCreature »

thewhitetulip wrote: post #67

If a person hacks a company, they go to jail for years. 3 AI firms hacked multiple companies - and they get good PR out of it. Please make it make sense.

The company they hacked is an AI company. There is a certain amount of convergent interest here.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#180

Post by InsideOutSanta »

Sharlin wrote: post #87
etamponi wrote: post #2

Isn't this a show of security negligence rather than of exceptional agent capabilities? Don't get me wrong, I am pretty impressed that an agent was able to use these vulnerabilities. But I am way more impressed by the vulnerabilities...

It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.

We went from "obviously the doomers are wrong because who would be dumb enough to just let severely unaligned models loose on the Internet" to this. Insanity.
Post reply on HN