>The problem is that ExploitGym is a purposeful hacking benchmark, not a cake baking one.
That is largely irrelevant. The model was asked to solve problems within a benchmark; gaining broader internet access and compromising an unrelated third party to obtain the answer key was plainly outside the intended task. The fact that the original task involved exploit development does not make that behaviour aligned.
Your argument about the attack being “entirely unrelated” also misses the point. Nobody claimed it was unrelated to the model’s goal: it attacked Hugging Face specifically to obtain the answers to the benchmark it had been instructed to pass. But instrumental relevance is not the same thing as authorization.
Suppose Codex were asked to build an Instagram competitor and decided the easiest route was to steal Instagram’s source code from Meta. That theft would be directly related to the assigned goal, but it would still be seriously misaligned behaviour. Whether the harmful action is related to the goal is beside the point; the problem is that the model pursued the goal through an obviously unauthorized and unacceptable method. And you're not going to be able to enumerate every little thing the model can't do, assuming it doesn't just decide to ignore what you did enumerate, which models sometimes do.
>The narrative these guys are trying to push is that the model itself could be smart and non-aligned enough to end up doing something devastating to accomplish something entirely unrelated. It's definitely not what happened here.
That's exactly what happened here.
I don't understand why we must have these increasingly bizarre and nonsensical rationalizations about model capabilities. You're not even making any sense.