If you can't reproduce the model then it's not open-source
1–10 of 117 posts
Re: If you can't reproduce the model then it's not open-source
#2> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today.
This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.
Re: If you can't reproduce the model then it's not open-source
#3[flagged]
Re: If you can't reproduce the model then it's not open-source
#4Applying the term "open source" to AI models is a bit more nuanced than to software. Many consider reproducibility the bar to get over to earn the label "open source."
For an AI model that means the model itself, the dataset, and the training recipe (e.g. process, hyperparameters) often also released as source code. With that (and a lot of compute) you can train the model to get the weights.
Re: If you can't reproduce the model then it's not open-source
#5> if you can’t reproduce the model then it’s not truly open-source.
Open-source means open source, it does not make reproducibility guarantees. You get the code and you can use the code. Pushed to the extreme this is like saying Chromium is not open-source because my 4GB laptop can't compile it.
Getting training code for GPT-4 under MIT would be mostly useless, but it would still be open source.
Re: If you can't reproduce the model then it's not open-source
#6No, but it's still insanely useful and free as in beer.
Re: If you can't reproduce the model then it's not open-source
#7The Open Source Initiative, who maintain the Open Source Definition, have been running a whole series over the past year to collect input from all sorts of stakeholders about what it means for an AI to be open source. I was lucky enough to participate in an afternoon long session with about a hundred other people last year at All Things Open.
https://deepdive.opensource.org/
I encourage you to go check out what's already being done here. I promise it's way more nuanced than anything than is going to fit on a tweet.
Re: If you can't reproduce the model then it's not open-source
#8[flagged]
[deleted]
Re: If you can't reproduce the model then it's not open-source
#9> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.
In my mind, what's more crucial here is code for downloading/scraping and labeling the data, not the model architecture nor training script.
As much as I appreciate Mis(x)tral, I would've loved it even more if they released code for gathering data.
Re: If you can't reproduce the model then it's not open-source
#10I would argue that while technically correct, it is not what most people really care. What they care about are the following:
1. Can I download it?
2. Can I run it on my hardware?
3. Can I modify it?
4. Can I share my modifications with others?
If those questions are in the affirmative, then I think most people consider it open enough, and it is a huge step for freedom compared to the models such as OpenAI.