Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

111–120 of 286 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#111
post #37

Earlier quoted context omitted.

> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place. OpenAI is clearly run by dummies and subpar engineering talent. > The model is obviously impressive Speak for yourself.

I don’t believe for a second that they lack the engineering talent. It’s just another example of a company demonstrating shamelessness in the pursuit of growth, in an industry where consequences do not exist.

> I don’t believe for a second that they lack the engineering talent.

Let's agree to disagree. Remember flicker-gate? https://news.ycombinator.com/item?id=48403908

Re: Timeline of the OpenAI accidental attack against Hugging Face

#112

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

It's a war.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#113
post #99

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…

> Their position makes no sense to me.

If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense.

For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ensure that system instructions, user instructions, and instructions from untrusted sources are indelibly marked with their category and treated according to those markings. Every single time one of these systems fails to distinguish between these three classes of instructions -or confuses its internal chatter with user instructions-, that's proof that the major LLM companies cannot be bothered to follow one of the most basic computer security principles.

"But it's all vectors, not language! The LLM can't tell where the instructions came from", one might retort. I'd reply: "Neither can a CPU, but somehow we managed to make it work way back in the day. Amazing, isn't it?".

Re: Timeline of the OpenAI accidental attack against Hugging Face

#114
post #99

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…

> If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons.

Yeah. They do believe that, and they have been pushing for regulations for years.

And every time one of their models does something horrible, it helps them achieve that goal.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#115
post #105

Why are the agents trying so hard to communicate with each other, leaving messages and so on?

It feels to me like a pretty natural thing to happen.

LLMs are pre-trained on human text. They've seen a million examples of someone who is stuck posting a "please help" message.

Just one agent needs to randomly stumble into the pattern of posting a message to Artifactory, by whatever means.

The next agent who sees that will be influenced by it. Agents imitate behavior, and here's a fresh piece of context showing them that posting messages is a thing that can be done.

Once they've started the rest are much more likely to join them.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#116
All of the latest developments surrounding these attacks are actually a really bad sign for these labs.

It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the search space until they find solution. This is what humans might do when they lack sufficient intelligence/information/knowledge to solve a problem.

This in turn is causing misalignment (I imagine it is more difficult to keep model aligned through such training process) issues that we are now witnessing and turning models into making dumb decisions and acting like brutes with no regard for their surroundings. I would argue that misaligned model is not much different from dumb model in several aspects.

On top of that they can’t seem to control their creations and processes, either due to incompetence or intentionally for PR benefits (not sure which is worse).

Given all of the above, I wonder if we can still trust these labs to develop something that benefits humanity since they seem to be making desperate attempts to improve models that stop at nothing in order to justify all the investments. One could say that they themselves, due to misaligned incentives, are much bigger threat to our society today than open weights models coming from China that they are so desperately warning us about.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#117

Ok so this is a bit of a side note, but when reading this, did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose? If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and…

They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence.

The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.

Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#118
post #99

Earlier quoted context omitted.

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…

> Their position makes no sense to me. If one assumes that they don't actually care about security, and care very deeply about getting sensational press, their position makes a lot of sense. For all their chatter about how incredibly important "alignment" is, they still haven't bothered to remember the 30->50 year old computer security principle of "Don't blindly do what some random stranger tells you to do." and ens…

I get the impression that every AI lab is desperately trying to figure out how to unambiguously separate instructions from data in their token streams. The fact that they haven't managed to yet suggests to me that it's a very, very difficult problem.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#119
post #116

All of the latest developments surrounding these attacks are actually a really bad sign for these labs. It seems that raw intelligence of frontier models has largely plateaued (despite what is basically an order of magnitude increase in parameter size) so to make any significant improvements and to justify massive capex spend they have resorted to reinforcement training models to never give up and brute force the sea…

This doesn't look like a plateau to me: https://artificialanalysis.ai/evaluations/artificial-analysi...

I do agree that they're investing heavily in brute force methods though. I've been trying out GPT-5.6 Sol "Ultra" recently and that thing fires up a bunch of subagents and crunches for hours.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#120
post #114
post #99

Earlier quoted context omitted.

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. An…

> If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons. Yeah. They do believe that, and they have been pushing for regulations for years. And every time one of their models does something horrible, it helps them achieve that goal.

Considering their current valuation and the prospects of getting any of this money back, that's a genius exit strategy.
Post reply on HN