> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by r…
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of op…
Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.
So what do we do about it?
1) We can accept it, and hope that the good guys AI can defend.
2) We can try to limit the access to it (AI proliferation?)
3) We stop the development of it
4) We can accept the risk and do nothing.
None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:
1) Nuclear triad / iron dome / early warning systems
2) Nuclear anti-proliferation treaties.
3) Dead Physicists
Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.
If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.