Earlier quoted context omitted.
Definitely a risk, and already happening, but I presume mostly closed source AIs are used for this? Like, people using the ChatGPT APIs to generate spam; or Grok just doing its normal thing. Don't see how the open vs closed debate has much to do with it.
You can't see how a hosted private model (that can monitor usage and adapt mechanisms to that) has a different risk profile than an open weight model (that is unmonitorable and becomes more and more runnable on more and more hardware every month)? One can become more controlled and wrangle in the edge-cases, and the other has exploding edges. You can have your politics around the value of open source models, but I fi…
The other assumption is that nefarious actors will care about any of this. They'll use what's available, or make their own models, or maybe even steal models (if China had an incredible AI, don't you think other countries would be trying to steal the weights?). Bad actors don't care about moral positions, strangely enough.