The unbearable cheapness of open weight models

This study seems to show that there are places where synthetic data, especially related to common crawl.

> Pure synthetic data remains non-advantageous over CC; notably, models trained on pure rephrased synthetic data will underperform those trained on CC at larger models.

But the tradeoffs seem to be different at large scale.

> Overall, these model scaling results suggest synthetic data appears comparably less favorable for pre-training larger LMs relative to its utility in data scaling scenarios. Despite outperforming training on CC, larger models are not as tolerant to a higher ratio synthetic data as larger data budgets. This observation aligns with practices where synthetic data is effective for smaller LMs or specific pre-training phases, but less predominantly used for the largest models.

How I am reading it is there are places where it is useful:

> Notably, any mixture involving synthetic data, or pure synthetic data (except pure QA), is projected to achieve a lower irreducible loss than training only on CommonCrawl.

But it also seems that on textbook scale synthetic data, they did show model collapse vs rephrased data.

> These results contribute mixed evidence on “model collapse" during large-scale single-round (n=1) model training on synthetic data–training on rephrased synthetic data shows no degradation in performance in foreseeable scales whereas training on mixtures of textbook-style pure-generated synthetic data shows patterns predicted by “model collapse".

IMHO there are some very specific areas where we aren't "data limited", like math, but as your reference states "Our work demystifies synthetic data in pre-training, validates its conditional benefits, and offers practical guidance."

Note the cost of 30% of the total dataset being synthetic, where the model starts amplifying the generator's biases, leading to a permanent degradation in downstream zero-shot capability on unseen out-of-domain natural tasks.

My takeaway is there is nuance where synthetic data is an amplifier and where it is a problem, and in my mind that paper demonstrates it will not solve the data problem in general.

That data includes not only the "weights" but also various files with required information, e.g. the tokenizer, the chat template, files that describe the structure of the "weights", e.g. number of layers, the number of "experts", routing information, etc. All this information may be distributed in many files (e.g. *.safetensors files with weights, *.json files etc.) or it may be aggregated in a single container file (with the .gguf extension).

You can see an example of the files included in a very simple open weights LLM here:

https://huggingface.co/google/gemma-4-12B-it/tree/main

Bigger LLMs have much more files, especially much more *.safetensors files, which contain the "weights". The "weights", i.e. matrices of numbers that are used in the computational algorithm that generates the output tokens, constitute the bulk of the data needed to run a model, i.e. from a few gigabytes to a couple of terabytes, which is why the term "open-weights" is used, but in fact by this term it is understood that all data needed for running inference is open.

For an open weights LLM, you do not have access to the data set used for training the model or to the algorithms that have been used during the training of that model.

You can still do some fine-tuning of the model, using your own training methods and your own additional training data. To facilitate this, several open weights models offer not only a model version that can be used for inference to implement a chat application or an agentic workflow, but also a "base" or "raw" version that is not suitable for being used directly for inference but which is suitable for you to do a post-training/fine-tuning, to create a model more appropriate for your particular needs.

An "open weights" model is sufficient for most of the potential LLM users, because training a model is something that requires expertise, expensive hardware and a lot of time, so few would be able to do it even when given access to the necessary data.

Computers and the internet did a lot to make production easier in addition to distribution. Anyone today can use a photo editor to superimpose text over an image in any font in seconds like it's child's play. That used to require knowledge of calligraphy. Film production used to require very expensive equipment that everyone now has built into their phone.

> Those prior technological changes never inverted a bottleneck, and this one does.

Before the printing press, copying books had to be done by hand. If you wanted a million copies of something made you had to be the church or a government. Today there are independent pundits who get a million impressions on their shitposts, and that's with consolidated platforms being largely against them.

We still have an entire edifice (copyright) which is structured around copying requiring a sufficiently centralized apparatus to serve as a useful chokepoint for imposing restrictions and collecting royalties, which is correspondingly under increasing distress as