Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Weights isn't Open Training

workshoplabs.ai

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Open Weights isn't Open Training

#21

Post by kazinator »

scottlamb wrote: post #5
Wowfunhappy wrote: post #3

Earlier quoted context omitted.

But how useful is source code if it takes millions of dollars to compile? At that point, if you do need to make changes, it probably makes more sense to edit the precompiled binary. Even the original developers are doing binary edits in most cases. I agree that open weight models should not be considered open source, but I also think the entire definition breaks down under the economics of LLMs.

There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? wh…

Security audits, etc, are possible because binary code closely implements what the source code says.

In this case, you have no idea what the weights are going to "do", from looking at the source materials --- the training data and algorithm --- without running the training on the data.

Re: Open Weights isn't Open Training

#23

Post by maxwg »

oscarmoxon wrote: post #2

The framing here is undersold in the broader discourse: "open weights" is a ruse for reproducibility. What you have is closer to a compiled binary than source code. You can run it, you can diff it against other binaries, but you cannot, in any meaningful sense, reproduce or extend it from first principles. This matters because OSS truly depends on the reproducibility claim. "Open weights" borrows the legitimacy of op…

The training methods are largely published in their open research papers - though arguably some open weight companies are less open with the exact details.

Realistically a model will never be "compiled" 1:1. Copyrighted data is almost certainly used and even _if_ one could somehow download the petabytes of training data - it's quite likely the model would come out differently.

The article seems to be talking more about the difficulties of fine tuning models though - a setup problem that likely exists in all research, and many larger OSS projects that get more complicated.

Re: Open Weights isn't Open Training

#24

Post by mschuster91 »

iamcreasy wrote: post #19
mschuster91 wrote: post #8

"open training" is something that won't ever happen for large scale models. For one, probably everyone's training datasets include large amount of questionable material: copyrighted media first and foremost (court cases have shown that AI models can regurgitate entire books almost verbatim), but also AI slop contaminating the dataset, or on the extreme end CSAM - for Grok to know how the intimate bits of children loo…

> "open training" is something that won't ever happen for large scale models https://www.swiss-ai.org/apertus Source: EPFL, ETH Zurich, and the Swiss National Supercomputing Centre (CSCS) has released Apertus, Switzerland’s first large-scale open, multilingual language model — a milestone in generative AI for transparency and diversity. Trained on 15 trillion tokens across more than 1,000 languages – 40% of the data…

I wasn't aware of that one, thanks.

Should have been more clear in my wording though - I was referring to commercially useful models.

Re: Open Weights isn't Open Training

#25

Post by cat_plus_plus »

Well, it's open training in the sense that the code is open source and you are free to fix it so it trains successfully. That's consistent with how open source works generally. In my experience unsloth is where new model training is usually fixed first.

Re: Open Weights isn't Open Training

#27

Post by alansaber »

maxwg wrote: post #23
oscarmoxon wrote: post #2

The framing here is undersold in the broader discourse: "open weights" is a ruse for reproducibility. What you have is closer to a compiled binary than source code. You can run it, you can diff it against other binaries, but you cannot, in any meaningful sense, reproduce or extend it from first principles. This matters because OSS truly depends on the reproducibility claim. "Open weights" borrows the legitimacy of op…

The training methods are largely published in their open research papers - though arguably some open weight companies are less open with the exact details. Realistically a model will never be "compiled" 1:1. Copyrighted data is almost certainly used and even _if_ one could somehow download the petabytes of training data - it's quite likely the model would come out differently. The article seems to be talking more abo…

Yes the issue is they can embelish the shit out of the papers b/c we only see the final result

Re: Open Weights isn't Open Training

#30

Post by vova_hn2 »

scottlamb wrote: post #5
Wowfunhappy wrote: post #3

Earlier quoted context omitted.

But how useful is source code if it takes millions of dollars to compile? At that point, if you do need to make changes, it probably makes more sense to edit the precompiled binary. Even the original developers are doing binary edits in most cases. I agree that open weight models should not be considered open source, but I also think the entire definition breaks down under the economics of LLMs.

There are lots of reasons to read through source code you never edit or recompile: security audits, interoperability, learning from their techniques, etc. And I think many of those same ideas apply to seeing the training data of a LLM. It will help you understand quickly (without as much experimentation) what it's likely to be good at, where its biases may be, where some kind of supplement (transfer learning? RAG? wh…

> security audits

If you are unable to run the multimillion training, then any kind of security audit of the training code is absolutely meaningless, because you have no way to verify that the weights were actually produced by this code.

Also, the analogy with source code/binary code fails really fast, considering that model training process is non-deterministic, so even if are able to run the training, then you get different weights than those that were released by the model developers, then... then what?

Post reply on HN