Even if you have open training, the corpus were compiled from millions of sources, labeled by experts manually, by AI, by outsourcing, etc. Is it "open by first principle" ?
Open Weights isn't Open Training
31–40 of 40 posts
Re: Open Weights isn't Open Training
#32The framing here is undersold in the broader discourse: "open weights" is a ruse for reproducibility. What you have is closer to a compiled binary than source code. You can run it, you can diff it against other binaries, but you cannot, in any meaningful sense, reproduce or extend it from first principles. This matters because OSS truly depends on the reproducibility claim. "Open weights" borrows the legitimacy of op…
> "Open weights" borrows the legitimacy of open source
I don't really see how open-weights models need to borrow any legitimacy. They are valuable artifacts being given away that can be used, tested and repurposed forever. Fully open models like the OLMo series and Nvidia's Nemotron are much more valuable in some contexts, but they haven't quite cracked the level of performance that the best open-weights models are hitting. And I think that's why most startups are reaching for Chinese base LLMs when they want to tune custom models: the performance is better and they were never going to bother with pretraining anyway.
Re: Open Weights isn't Open Training
#33"open training" is something that won't ever happen for large scale models. For one, probably everyone's training datasets include large amount of questionable material: copyrighted media first and foremost (court cases have shown that AI models can regurgitate entire books almost verbatim), but also AI slop contaminating the dataset, or on the extreme end CSAM - for Grok to know how the intimate bits of children loo…
I’m not convinced that Grok’s dataset must contain CSAM for it to generate CSAM. Surely a combination of nude adults and clothed children would allow for it to synthesize CSAM? (Disclaimer: I’m not in favor of AI in general and definitely not in favor of what Grok is doing specifically. I’m just entirely sold on the claim that its dataset must contain CSAM, though I think it is probably likely that it has at least so…
How many llamas riding motorcycles are in the dataset for it to be able to generate images for that? Why does that not extend to CSAM?
Re: Open Weights isn't Open Training
#34Open-weight AI is actually analogous to closed source, free shareware you can decompile and modify yourself and run on your computer or a cloud server of your choice.
It's a clear distinction to proprietary AI, which is analogous to SaaS software controlled by a company that runs it on its own cloud, and owns your data.
But it's still not open source.
Re: Open Weights isn't Open Training
#35Earlier quoted context omitted.
There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? wh…
> security audits If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code. Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get…
> considering that model training process is non-deterministic
Why would it have to be? Just use PRNG with published seeds and then anyone can reproduce it.
Re: Open Weights isn't Open Training
#36Earlier quoted context omitted.
> security audits If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code. Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get…
> considering that model training process is non-deterministic Why would it have to be? Just use PRNG with published seeds and then anyone can reproduce it.
I have zero actual experience in training models, but in general, when parallelizing work: there can be fundamental nondeterminism (e.g., some race conditions) that is tolerated, whose recording/reproduction can be prohibitive performance-wise.
Re: Open Weights isn't Open Training
#37The picture on that blog post is very cool. It gives Hetch Hetchy vibes. Is there software to convert a photo into ASCII art?
Re: Open Weights isn't Open Training
#38Earlier quoted context omitted.
There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? wh…
> security audits If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code. Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get…
If you're going through the effort to be open source you can probably set up fixed batch sizes and deterministic combination of batches without too much more effort. At least I hope it's not super hard.
Re: Open Weights isn't Open Training
#39Earlier quoted context omitted.
> "open training" is something that won't ever happen for large scale models https://www.swiss-ai.org/apertus Source: EPFL, ETH Zurich, and the Swiss National Supercomputing Centre (CSCS) has released Apertus, Switzerland’s first large-scale open, multilingual language model — a milestone in generative AI for transparency and diversity. Trained on 15 trillion tokens across more than 1,000 languages – 40% of the data…
I wasn't aware of that one, thanks. Should have been more clear in my wording though - I was referring to commercially useful models.
Apertus is available for commercial use based on its license. It doesn't produce state-of-the-art (SOTA) results, but for many organizations, it greatly reduces risk of copyright infringement and even if it does, there is a direct way to address it. In fact, if you were posting in good faith, I would expect people very concerned about questionable training data to be more aware of Apertus.
Re: Open Weights isn't Open Training
#40Earlier quoted context omitted.
There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? wh…
> security audits If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code. Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get…
I probably shouldn't have led with that example because yeah, reproducible (and cheap) builds would be best for security audits. But I wouldn't say it's absolutely meaningless. At least it can guide your experimentation, and if results start differing radically from what you'd expect from the training data, that raises interesting questions.