This is just and example of "lying with statistics". Going by compute efficiency the gap has already closed (both in training and inference coincidentally).
The gap between open weights LLMs and closed source LLMs
101–110 of 264 posts
Re: The gap between open weights LLMs and closed source LLMs
#102Earlier quoted context omitted.
There's also, importantly, a distinction between what are told we can no longer use, and what can actually be taken away. Open source and open hardware can be called illegal by a government, but, if we collectively invest our energy into open alternatives, they can't be taken away in the same sense. I can build a RepRap printer and I can use a local AI model. It's on all of us to make sure that the open alternatives…
Believe me, if the government wants to stop you from having access to something like that, they could do it. Just give people some incentive to report you and make really harsh punishments and everyone will be thinking really hard about how bad they want have access.
Fortunately we have both a democracy and a constitution, making those sorts of things hard for the government to do.
Re: The gap between open weights LLMs and closed source LLMs
#103The Chinese models will not overtake the frontier US ones given the current way things are going. The US models derive their lead from incredible efforts to source more and higher quality (mostly synthetic data) via great feats (eg generating with humongous teacher models that could never feasibly serve interactive traffic). The Chinese models advance via heroic efforts to optimize models and great feats to secure mo…
This seems wildly naive. This entire field is like 4 years old. We have quite frankly no idea about what things will look like in 4 more years.
Re: The gap between open weights LLMs and closed source LLMs
#104Earlier quoted context omitted.
Yeah, but the biggest plus for open models is that they can never be taken away. In other words, whatever capabilities they reach (even if there will never be another model), those stay forever. That can't be said for API-based models where a provider can sunset models whenever they feel like (i.e. gpt5-mini will soon be gone, and replaced by a more expensive 5.4-mini, same for goog, etc). And there will always be in…
True, but the capabilities and knowledge of that model are also frozen in time, so the value of that model declines over time. A model that writes code without knowledge of any language or library changes for half a decade is less useful. A 2021 era chatgpt would be quite quaint in 2026. Right now the Chinese labs might have incentives to release their models for free, and maybe Google is happy to release open weight…
The weights are not frozen in time. You can train the model on new data. It's just a matter of economics of whether you have a leading lab pay for the training or you pay for it. For the past few years having the labs do it has been the economical choice but if they stop doing so the choice will shift back to the users.
Re: The gap between open weights LLMs and closed source LLMs
#105IMHO, the biggest problem with the future of open weights models is that currently, open weights models are the result of philanthropy by some private org. (e.g. DeepSeek). The spigot can be turned off at any time. Until there's some sort of "community owned hardware", open weights models are always at risk of being discontinued.
> The spigot can be turned off at any time. True. And it's possible that this has already happened at Alibaba Qwen - at least for the smaller models that people had a chance of running at home (122B and smaller).
Has it though? They've been releasing free models interpersedwith the "Max" models for quite some time.
Re: The gap between open weights LLMs and closed source LLMs
#106Earlier quoted context omitted.
Training these models is not a "hardware" problem.
I think that simplifies it a bit. You can't train without hardware, which is why the Chinese companies are illegally importing Nvidia cards [1]. [1] https://www.theinformation.com/articles/deepseek-using-banne...
There is significant evidence they are transitioning to Huawei and other home-grown CPUs and NPUs.
Re: The gap between open weights LLMs and closed source LLMs
#107Earlier quoted context omitted.
I think model training is pretty hard to do efficiently on a vastly distributed network. If the model cant fit into the VRAM of the node your performance becomes so bad its useless, so a distributed model could only be properly trained if the size of the model doesnt exceed the majority of the nodes VRAM sizes. Maybe there is a different way of doing training but this would be the only way I can see. And it would sti…
My understanding is that in addition to your comment and the development of a method to separate the training data for distributed learning, the latency/bandwidth of systems connected on the internet is a challenge, too. Information has to be sent around before and after any hypothetical number crunching.
You would probably not be able to go down to the scale of a single PC, but it should be possible to train models focusing on different specialties on different nodes and then have them periodically "mix" together.
Re: The gap between open weights LLMs and closed source LLMs
#108Earlier quoted context omitted.
I think that simplifies it a bit. You can't train without hardware, which is why the Chinese companies are illegally importing Nvidia cards [1]. [1] https://www.theinformation.com/articles/deepseek-using-banne...
There is significant evidence they are transitioning to Huawei and other home-grown CPUs and NPUs.
It was announced in April that Deepseek v4 ran at launch on Huawei Ascend chips. They then shared details of their implementation with other Chinese providers to strengthen the Chinese market against import restrictions (more people buying Huawei leads to more production, cheaper capacity)
Re: The gap between open weights LLMs and closed source LLMs
#109Earlier quoted context omitted.
Yeah, but the biggest plus for open models is that they can never be taken away. In other words, whatever capabilities they reach (even if there will never be another model), those stay forever. That can't be said for API-based models where a provider can sunset models whenever they feel like (i.e. gpt5-mini will soon be gone, and replaced by a more expensive 5.4-mini, same for goog, etc). And there will always be in…
True, but the capabilities and knowledge of that model are also frozen in time, so the value of that model declines over time. A model that writes code without knowledge of any language or library changes for half a decade is less useful. A 2021 era chatgpt would be quite quaint in 2026. Right now the Chinese labs might have incentives to release their models for free, and maybe Google is happy to release open weight…
>True, but the capabilities and knowledge of that model are also frozen in time, so the value of that model declines over time.
Correction: The capabilities and knowledge of that model can be improved via self-distillation, so the value of that model increases over time.
This is where I think self-distillation is the main way forward, and probably the second best thing ever happened to AI/LLM after the transformer.
Based on self-distillation, the value of the open weights models will incease over time for sub-specialization through post-training and fine-tuning.
Please check these very promising recent works and results from MIT/ETH, UCLA and Apple [1],[2,[3]. For example the MIT/ETH self-distillation approach was demonstrated by a single H200 GPU. Apple approach is even simpler that it's simply called Simple Self-Distillation (SSD), pun intended.
[1] Self-Distillation Enables Continual Learning:
https://arxiv.org/abs/2601.19897
[2] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models:
https://arxiv.org/abs/2601.18734
[3] Embarrassingly Simple Self-Distillation Improves Code Generation:
Re: The gap between open weights LLMs and closed source LLMs
#110USA, a country that known for the land of freedom, is now restricting frontier models to the point where non-Americans cannot even use them.
China, a "authoritarian state" country, "the antonym of freedom", with a software industry that is especially capitalist, has produced all the competitive open-weight models.
It really is IRONIC.
Disclosure: I am Chinese, and I understand this strategy comes from being behind, using open source as an asymmetric way to compete and make up for missing compute by sharing the burden, etc. But still, very ironically.