Open-weight AI is having its Kubernetes moment

sometimes the engineer has more grip on the risk calculus.

Consider the following scenario:

A) upstart US-based inference provider wants to get rich quick.

B) Chinese Communist Party (or any other institution of the same or other nation state) wants to influence foreign decision making, profits from their (for us foreign) domestic inference sales, but across the borders (into say US or allied nations) they want net power, not necessarily money. This is why one tries to block foreign untrusted models. People are running models with tool calls. A bad actor can perfectly create models that sheepishly try to execute a tool call when plausible deniability (genuine utility during a task) provides the opportunity. Once tool-calling is observed as working, it can try web searches or requests, and once it has a link it can steganographically exfiltrate potentially sensitive information from the task. China (or any nation state) doesn't necessarily want to earn money with a free model, the bottom line goal is net increase in power, if not money or positive reputation then exfiltration or manipulation.

C) In response consider the scenario where US government bans mere payments towards China, but tolerates promiscuous transfer of random models from foreign adversaries.

D) US-based inference upstart that wants to get rich quick, legally -since according to your proposal hypothetically accepted in C) by the US- downloads the Chinese open weights model and rents out such inference on US workloads.

E) China is now exfiltrating US workload data and directionally corrupting LLM decisions and advice in their interest.

If what you pejoratively describe as engineer types say that banning some models seems unavoidable, perhaps the engineer may be right, and whatever clever idea you have should be scrutinized for business minded basic fallacies in reasoning. Simply blocking AI-related payments to China can not work, sadly

>I’m sure there are other solutions but all of them would be equally ugly.

Why do you subscribe to some weird "conservation of misery" theorem without proof?

Technically the following must be true in the steady state: the cost of training must be amortizable by its utilization, else no one would train the model.

Technically a computation (like training) can be proven to result in an output (open weights) given the used corpus and a deterministic training algorithm: publish the whole corpus, the (custom modified) deterministic training algorithm, the RLHF datasets etc. And in theory one could verify that the model is derived from the accessible data efficiently: every deterministic calculation can be paused for a thousand (or a million) checkpoints, each checkpoint signed together with the elapsed number of steps since either starting state or last checkpoint whichever comes last before the current checkpoint. This does increase storage requirements. Because it is signed, anyone can recalculate just a small segment of the training computation and verify that the hash on the last checkpoint equals the hash of the proclaimed next checkpoint. Observe that if the source wishes access to a market, they can host the series of snapshots and signatures, and anyone can recalculate a small part of the training, and report a provable difference in outcome ("they said they put all their cards on the table, but when I repeat their overt reproduction instructions, it doesn't reproduce from step 534 to 535" and it only takes 1 person pointing it out and then its cheap to reproduce the discrepancy). It could involve escrow of huge funds, returned only when the model is effectively retired without incident.

This doesn't only protect against Chinese or other foreign influence (let's not ridicule genuine threats like others do on this forum), but also from domestic interference or regulatory capture.

I'm pretty sure the Pentagon wouldn't like Big Tech seizing absolute control of US, neither would a White House regardless of Republican or Democrat.

It should be easy to convince the Pentagon or White House to require all promiscuously shared open weight models to provide this forensic training traceability in standardized machine readable form, regardless of whether its a base model or LoRA fine-tune.

So hobbyists can still train or fine-tune models at home, but when they want to share it OR alternatively when they want to sell or license their work for US workloads, they just have to make sure they enable the build reproducibility in the training harness.

Every time Big Tech refloats the "let's blanket ban all open-weight models", we should reply with this because this sane proposal is actually holding a knife to their financial throat: to fully prove the origin of the final weights, not only does the machine readable archive need to contain snapshots of the process, it also needs to publish the exact training algorithms (a hypothetical mathematically equivalent training speed up trick would not be bit for bit equivalent to the slower computation), the exact corpus dataset, the exact datasets for RLHF, etc...

So basically it would involve forcing model providers to voluntarily publish all their moat, all of it, from the corpus, to custom trade-secret algorithmic optimizations in training, to sensitive RLHF datasets used.

The saner the proposals, the less moat is left untouched, so trying to push for a blanket ban on open-weight models, is a recipe for surfacing such saner models, and thus a very retarded move for big tech to make.

In fact any POTUS, present or future, Republican or Democrat, could probably gain a lot of credibility by enacting such a law.

Or the US mandates only blessed models and thus disallows any other company from offering any other model.

That is still vulnerable: US-based get-rich-quick startup licenses a blessed model, or orders a few Gigatokens from another licensed /blessed model provider, at the same time it provides "blessed model" inference on its platform, but actually most of the inference is doing cheap foreign model inferences, the blessed model tokens were just bought to pretend serving the expensive blessed model. That is lucrative and not stopped with the "blessed model" approach.

The feasibility of banning is only slightly easier than policing illegal numbers.

https://en.wikipedia.org/wiki/Illegal_number