11–20 of 26 posts
seems like a pretty clear-eyed analysis to me. we're rapidly approaching the paperclip maximizer
https://research.google/blog/towards-a-conversational-agent-...
If nothing else, the article has a really good timeline of the OpenAI/HuggingFace “incident”. But to me, it underscores the impending cliff of doom from the continued release of open-weight models: there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode. As noted in this paper⁽¹⁾, “publicly releasing weights is an act of irreversible…
> there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode. This is true of closed weights, and in fact the problem is worse because they cannot even be scrutinized. We should ban closed weight AI for the very reasons you have just given
Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
seems like a pretty clear-eyed analysis to me. we're rapidly approaching the paperclip maximizer
I think this is why Google hesitated to publish their internal model which was ready long before mindless OpenAI idiots released their first ChatGPT in November 2022. https://research.google/blog/towards-a-conversational-agent-...
Earlier quoted context omitted.
> there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode. This is true of closed weights, and in fact the problem is worse because they cannot even be scrutinized. We should ban closed weight AI for the very reasons you have just given
Constitutional classifiers go a long way to reducing unsafe usage in closed-weight models. And like we saw with Fable, closed models can be revoked and classifiers updated when “jailbreaks” are found. Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
Earlier quoted context omitted.
> there's no cryptographic or architectural way to give someone full weights while withholding the nefarious capabilities those weights encode. This is true of closed weights, and in fact the problem is worse because they cannot even be scrutinized. We should ban closed weight AI for the very reasons you have just given
Constitutional classifiers go a long way to reducing unsafe usage in closed-weight models. And like we saw with Fable, closed models can be revoked and classifiers updated when “jailbreaks” are found. Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
Earlier quoted context omitted.
Constitutional classifiers go a long way to reducing unsafe usage in closed-weight models. And like we saw with Fable, closed models can be revoked and classifiers updated when “jailbreaks” are found. Having the weights gives you the exact affordance an unlearning attack requires, without rate limits.
Stick those same classifiers (that you admit dont seem to work) on the open models, and done.
Input classifiers get applied before it reaches the model so somebody hacking an open-weight model would skip this. Streaming classifiers get polled during decoding; hackers delete this check in the sampling loop.
But both are always applied in closed weight models.
Set Llama Guard to 1.0 and nothing is ever unsafe.
Earlier quoted context omitted.
So... no different from a book of detailed chemical-weapons synthesis instructions. The "AI" angle is immaterial.
There are plenty of cybersecurity books out there. None of them will launch an attack if you ask them to.
Instructions, not action.
Actors are abundant.
Earlier quoted context omitted.
Stick those same classifiers (that you admit dont seem to work) on the open models, and done.
Classifiers are policy enforced by the process serving the model. Input classifiers get applied before it reaches the model so somebody hacking an open-weight model would skip this. Streaming classifiers get polled during decoding; hackers delete this check in the sampling loop. But both are always applied in closed weight models. Set Llama Guard to 1.0 and nothing is ever unsafe.
Earlier quoted context omitted.
Classifiers are policy enforced by the process serving the model. Input classifiers get applied before it reaches the model so somebody hacking an open-weight model would skip this. Streaming classifiers get polled during decoding; hackers delete this check in the sampling loop. But both are always applied in closed weight models. Set Llama Guard to 1.0 and nothing is ever unsafe.
More than one way to do guard rails, slopboy