So, we need something like dockerfiles for models?
If you can't reproduce the model then it's not open-source
61–70 of 117 posts
Re: If you can't reproduce the model then it's not open-source
#62> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.
The GPL describes the source as the "preferred form for modification".
The model weights ARE the preferred form for modification
Re: If you can't reproduce the model then it's not open-source
#63I don't agree, and the analogy is poor. One can do the things he lists with a trained model. Having the data is basically a red herring. I wish this got more attention. Open/free software is about exercising freedoms, and they all can be exercised if you've got the model weights and code. https://www.marble.onl/posts/considerations_for_copyrighting...
My main concern is that if all you have are weights you're stuck hoping for the benevolence of whatever organization is actually able to train the model with their secret dataset.
When they get bought by Oracle and progress slows to a crawl because it's not profitable enough to interest them, you can't exactly do a LibreOffice. Or they can turn around and say "license change, future versions may not be used for " and now you're stuck with whatever old version of the model while they steamroll your project with newer updates.
Open weights are worth nothing in terms of long term security of development, they're a toy that you can play with but you have no assurances of anything for the future.
Re: If you can't reproduce the model then it's not open-source
#64> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.
No it’s not. You have everything you need to modify the models to your own liking. You can explore how it works.
This analogy is bad. Models are unlike code bases in this way.
Re: If you can't reproduce the model then it's not open-source
#65Earlier quoted context omitted.
Open Source and Free Software wasn't formulated to deal with the need for this level of gargantuan amounts of data and compute. Can the public compete? What percentage of the technical public could we expect to participate, and how much data, compute, and data quality improvement could they bring to the table? I suspect that large corporations are at least an order of magnitude advantaged economically.
The public doesn't have the resources to train the largest state-of-the-art LLMs, but training useful LLMs seems doable. Maybe not for most individuals but certainly for a range of nonprofits, research teams and companies.
But not nearly as in reach as modifying open source models.
Re: If you can't reproduce the model then it's not open-source
#66I don't agree, and the analogy is poor. One can do the things he lists with a trained model. Having the data is basically a red herring. I wish this got more attention. Open/free software is about exercising freedoms, and they all can be exercised if you've got the model weights and code. https://www.marble.onl/posts/considerations_for_copyrighting...
My main concern is that if all you have are weights you're stuck hoping for the benevolence of whatever organization is actually able to train the model with their secret dataset. When they get bought by Oracle and progress slows to a crawl because it's not profitable enough to interest them, you can't exactly do a LibreOffice. Or they can turn around and say "license change, future versions may not be used for " and…
Everything you just said applies to normal software. Oh no! Big Corp just started a closed fork of their open source codebase! Well, the open source version is still there. The open source community can build off of it.
You may complain that subsequent models are not iterative on the past and so having that old version doesn’t help; but then the data probably changes too so having the old data would largely leave you with the same old model.
Re: If you can't reproduce the model then it's not open-source
#67> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.
In my mind, what's more crucial here is code for downloading/scraping and labeling the data, not the model architecture nor training script. As much as I appreciate Mis(x)tral, I would've loved it even more if they released code for gathering data.
I don't mind to have ready to use datasets instead the code for downloading/scraping and labeling. It will save a lot of time. It is not complicated to write some code for gathering the data, it might be sometimes impossible to replicate the datasets after all if some parts of the data which you have to scrape are already gone (removed because of various reasons).
Re: If you can't reproduce the model then it's not open-source
#68Earlier quoted context omitted.
My main concern is that if all you have are weights you're stuck hoping for the benevolence of whatever organization is actually able to train the model with their secret dataset. When they get bought by Oracle and progress slows to a crawl because it's not profitable enough to interest them, you can't exactly do a LibreOffice. Or they can turn around and say "license change, future versions may not be used for " and…
Everything you just said applies to normal software. Oh no! Big Corp just started a closed fork of their open source codebase! Well, the open source version is still there. The open source community can build off of it. You may complain that subsequent models are not iterative on the past and so having that old version doesn’t help; but then the data probably changes too so having the old data would largely leave you…
When you train an updated model on a new dataset do you really start by deleting all of the data that you collected for training the previous version?
Re: If you can't reproduce the model then it's not open-source
#69We made our last language model fully reproducible including all datasets, training details, hyper parameters etc: https://stability.wandb.io/stability-llm/stable-lm/reports/S...
Re: If you can't reproduce the model then it's not open-source
#70Earlier quoted context omitted.
Everything you just said applies to normal software. Oh no! Big Corp just started a closed fork of their open source codebase! Well, the open source version is still there. The open source community can build off of it. You may complain that subsequent models are not iterative on the past and so having that old version doesn’t help; but then the data probably changes too so having the old data would largely leave you…
When you train an updated model on a new dataset do you really start by deleting all of the data that you collected for training the previous version?
Probably not. But if it’s the new data providing the advantage then you’re not exactly better off having the old data and the model vs. just having the model.