Someone who until yesterday did not seem bothered by his technology being possibly used to bomb elementary girls school in another country seems to suddenly care about the repression of citizens in yet another country. No, we don't buy your virtue signaling. And we certainly don't need your better-than-thou opinions on this year's "nightmare scenarios".
The elementary school example is a clear sign of bad faith. Neither Iran or the USA have intentionally blown up any school children.
Our position on open-weights models
721–730 of 1001 posts
Re: Our position on open-weights models
#722On Dario's concerns: It is very relevant whether frontier capabilities and research continue to be diffused in the open, because leveling the intelligence playing field empowers ordinary people more than it empowers governments that already have access to the frontier. Models that are trained specifically for military use by governments should not be open-sourced to prevent an arms race, but general intelligence is d…
A base model is absolutely not aligned. Pretraining doesn't teach an LLM to mimic the average human - doing so would make an LLM perform quite badly at predicting most of the dataset. It teaches it to mimic all possible humans¹, inferring what sort of persona to take depending on the context. A base model can indeed convincingly act like an "average person in the developed world", including by making the same sort of moral decisions that such a person would do... but it can also convincingly act like a shitty human, or like Hitler², or like any other actor that left its traces in the training dataset. A pretrained LLM therefore contains multitudes of personas, some of which would be considered aligned if you could make the LLM elicit them robustly, and most of them wouldn't be. But then you're left with the problem of how to make a base model elicit a very specific persona robustly even in out-of-distribution scenarios, which is not necessarily easier than solving alignment any other way.
(Another note is that the question of how aligned base models are is rather academic because almost nobody uses them anyway, because it's hard to get powerful capabilities by pretraining alone. Nowadays most of the frontier models' programming and math abilities are driven by RLVR.)
¹ Really "all generators of text that went into the training dataset".
² Even after RL training LLMs still retain those personals and can be convinced to elicit them quite easily, though it takes a tiny bit of finetuning: see https://arxiv.org/pdf/2512.09742.
Re: Our position on open-weights models
#723Re: Our position on open-weights models
#724Re: Our position on open-weights models
#725I can't remember the last time (if ever) a company managed to go from golden goose to.. whatever this is.. so quickly. The permanent defensiveness in his presentation is really hard to swallow, it actively puts me off wanting to believe in or rely on their product line with Dario at the helm. I don't even understand the logic leading up to this post. Who was it even hoping to convince. Is it possible Anthropic is due…
Anthropic has been surprisingly bad at comms for the last few months, which I find crazy in such a competitive market.
Re: Our position on open-weights models
#726In the first paragraph, > Anyone who has read my past writing should know that I don’t regard such bans as a useful measure, Later (on banning chip sales to china) > we should crack down on the rampant smuggling and workarounds used to obtain access to such chips. If you truly believe that bans don't work, the same applies to hardware too. Furthermore, Dario says later "To address these concerns, I do support the fol…
Bans on Chinese open weight models being used in the US and bans on AI chips and semiconductor manufacturing equipment being exported to China are two extremely different things, and it's not inconsistent in any way to oppose one and endorse another.
Re: Our position on open-weights models
#727Schrödinger's China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!) The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be…
But why call them overpriced? Compared to what? Even if we take the margin reports at face value, we don’t know their training costs, etc.
Curious if this was more of an emotional take or if there’s actual evidence behind it.
Re: Our position on open-weights models
#728this guy is smokin spaceballs and should be institutionalized.