Live data from Hacker News

Timeline of the OpenAI accidental attack against Hugging Face

simonwillison.net

241–250 of 293 posts

Re: Timeline of the OpenAI accidental attack against Hugging Face

#241
post #119

Earlier quoted context omitted.

This doesn't look like a plateau to me: https://artificialanalysis.ai/evaluations/artificial-analysi... I do agree that they're investing heavily in brute force methods though. I've been trying out GPT-5.6 Sol "Ultra" recently and that thing fires up a bunch of subagents and crunches for hours .

Here's the performance of frontier models without reasoning, to more directly address the claim that raw performance is plateauing: https://artificialanalysis.ai/evaluations/artificial-analysi... I don't have any insider info, but if model sizes actually have increased exponentially since GPT 4.1, there's an argument to be made that there are diminishing returns in scaling pretraining alone. Also interesting thing I…

The trend I've found most interesting is models of the same size getting better.

I'm very much looking forward to seeing how Qwen 3.8 27B compares to Qwen 3.6 27B next week, for example.

And the latest DeepSeek v4 Flash has extremely impressive performance for a 304B model.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#242
post #46

Norbert Wiener in 1960: "As is now generally admitted, over a limited range of operation, machines act far more rapidly than human beings and are far more precise in performing the details of their operations. This being the case, even when machines do not in any way transcend man's intelligence, they very well may, and often do, transcend man in the performance of tasks. An intelligent understanding of their mode of…

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

"Complete subservience and complete intelligence do not go together."

I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.

Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)

Re: Timeline of the OpenAI accidental attack against Hugging Face

#244
post #93

I think one of the most interesting details here might be tucked away in that first bulletin point: > May 7: OpenAI starts a new training run for an experimental, unreleased model. (Do they mean an evaluation run? They say training run in the video, and later mention a “reward signal to judge how well they’re doing”, so I guess this really was about training a model, not evaluating one that was already trained.) The…

Yes, the message boards and collaborative hacking occurring during training runs was BY FAR the biggest bombshell revealed, and OpenAI doesn't even seem to realize it. The fact that they continued the training runs, with those rewarded behaviors included, and didn't wind back training to before hand, shows that they fundamentally do not understand alignment and safety (somewhat interestingly, their previous head of s…

don't forget Altman's lies about dedicating resources to the alignment team

>Altman continued touting OpenAI’s commitment to safety, especially when potential recruits were within earshot. In late 2022, four computer scientists published a paper motivated in part by concerns about “deceptive alignment,” in which sufficiently advanced models might pretend to behave well during testing and then, once deployed, pursue their own goals. (It’s one of several A.I. scenarios that sound like science fiction—but, under certain experimental conditions, it’s already happening.) Weeks after the paper was published, one of its authors, a Ph.D. student at the University of California, Berkeley, got an e-mail from Altman, who said that he was increasingly worried about the threat of unaligned A.I. He added that he was thinking of committing a billion dollars to the issue, which many A.I. experts considered the most important unsolved problem in the world, potentially by endowing a prize to incentivize researchers around the world to study it. Although the graduate student had “heard vague rumors about Sam being slippery,” he told us, Altman’s show of commitment won him over. He took an academic leave to join OpenAI.

But, in the course of several meetings in the spring of 2023, Altman seemed to waver. He stopped talking about endowing a prize. Instead, he advocated for establishing an in-house “superalignment team.” An official announcement, referring to the company’s reserves of computing power, pledged that the team would get “20% of the compute we’ve secured to date”—a resource potentially worth more than a billion dollars. The effort was necessary, according to the announcement, because, if alignment remained unsolved, A.G.I. might “lead to the disempowerment of humanity or even human extinction.” Jan Leike, who was appointed to lead the team with Sutskever, told us, “It was a pretty effective retention tool.”

The twenty-per-cent commitment evaporated, however. Four people who worked on or closely with the team said that the actual resources were between one and two per cent of the company’s compute. Furthermore, a researcher on the team said, “most of the superalignment compute was actually on the oldest cluster with the worst chips.” The researchers believed that superior hardware was being reserved for profit-generating activities. (OpenAI disputes this.) Leike complained to Murati, then the company’s chief technology officer, but she told him to stop pressing the point—the commitment had never been realistic.

per https://www.newyorker.com/magazine/2026/04/13/sam-altman-may...

Re: Timeline of the OpenAI accidental attack against Hugging Face

#245
post #233

Earlier quoted context omitted.

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.” This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese…

> Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. Where did you see that number?

I knew when I wrote that it was a bare assertion, based partly on memory. This is an approximation based on a few sources, the principal of which was this article, which pulls from a bunch of other sources in turn.

https://www.secondtalent.com/resources/ai-trends-in-china/

Re: Timeline of the OpenAI accidental attack against Hugging Face

#246
post #236
post #231

Earlier quoted context omitted.

Why do these super agents need package managers anyway? Can’t they basically instantiate most OSS projects from scratch anyway? Spin up a sub agent to write me an OS interface in C. Done

This is part of the training process for a model. They're trying to train it to effectively use existing software to solve problems.

I see that now. I've been confused about that to this point, I guess. I understood this to be a specific infosec exercise.

[Edit: eh, a bit of both. They were doing RL on a hacking exercise. It hacked the harness which was plugged into the phone line. Same question.]

Re: Timeline of the OpenAI accidental attack against Hugging Face

#249

I wish we could stop sensationalizing this about the AI and really just understand the incompetence of the labs disabling an internet connection in a sandbox.

Hacker News doesn’t have the wherewithal to understand that this is just marketing by OpenAI.

Re: Timeline of the OpenAI accidental attack against Hugging Face

#250

Earlier quoted context omitted.

What a paper! And you missed an even MORE relevant excerpt!! Man and Slave The problem, and it is a moral prob- lem, with which we are here faced is very close to one of the great problems of slavery. Let us grant that slavery is bad because it is cruel. It is, how- ever, self-contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of ou…

"Complete subservience and complete intelligence do not go together." I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want. Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty w…

>I'm not convinced this is true. Perhaps for a human it is, but we can give an artificial mind whatever properties we want.

Just because it's artificial doesn't mean you can 'give it any properties you want'. We certainly can't do that for Deep ANNs.

>Even for people, what about e.g. the extremely intelligent military general who is absolutely loyal to his king? (Of course, some generals do lead coups and you can't know in advance which ones, but I'd think there are plenty who have undying loyalty, and I don't think it correlates to overall intelligence!)

Is there a human that is absolutely loyal under any condition? Would that general be loyal if the king asked him to slaughter his family ? What about if the king asked him to betray his most deeply held convictions ? Loyalty is a 2 way street.

Post reply on HN