Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you can't reproduce the model then it's not open-source

twitter.com

Post reply on HN
Print view View original

91–100 of 117 posts

Re: If you can't reproduce the model then it's not open-source

#91

Post by PeterisP »

fzliu wrote: post #9
ssgodderidge wrote: post #2

> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.

In my mind, what's more crucial here is code for downloading/scraping and labeling the data, not the model architecture nor training script. As much as I appreciate Mis(x)tral, I would've loved it even more if they released code for gathering data.

IMHO much of the key training data can't simply be downloaded/scraped/labeled, no matter what code you had - it's not like it's freely accessible to everyone and just needs some code to get it and process it. You can't scrape all of Google Books archive or all of Twitter, and quite a few things that could be scraped at one point may actively prevent you from scraping them now.

Re: If you can't reproduce the model then it's not open-source

#92

Post by Zuiii »

ssgodderidge wrote: post #2

> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.

Off-topic but that's why I always fail to pick up android dev after so many false starts. It just never felt right.

Android is not open source.

Re: If you can't reproduce the model then it's not open-source

#93

Post by tbrownaw »

> The “source code” for a work means the preferred form of the work for making modifications to it.

-- gplv3

These AI/ML models are interesting in that the weights are derived from something else (training set), but if you're modifying them you don't need that. Lots of "how to do fine-tuning" tutorials floating around, and they don't need access to the original training set.

Re: If you can't reproduce the model then it's not open-source

#94

Post by wruza »

declaredapple wrote: post #31
fzliu wrote: post #9

Earlier quoted context omitted.

In my mind, what's more crucial here is code for downloading/scraping and labeling the data, not the model architecture nor training script. As much as I appreciate Mis(x)tral, I would've loved it even more if they released code for gathering data.

I'm speculating they are attempting to avoid controversy about their datasources. That and a possible competitive edge depending on what specific sets/filtering they're using.

It’s common for third party model testers to not disclose what they mean by “Refusal” parameter as well, for obvious reasons. The world is full of witch-hunting maniacs now and will stay so for an indefinite amount of time. Just wait until the whole thing becomes more widely known and they realize. All AI companies have to hurry up before the doors shut.

Re: If you can't reproduce the model then it's not open-source

#95

Post by tonyarkles »

jncfhnb wrote: post #85
tonyarkles wrote: post #75

Earlier quoted context omitted.

> You have everything you need to modify the models to your own liking. What if I wanted to train it using only half of its training set? If the inputs that were used to generate the set of released weights are not available I can’t do that. I have a set of weights and the model structure but without the training dataset I have no way of doing that. To riff on the parent post, I have: Source + Compiler => Binaries Fo…

> Maybe I think the model author should have used a deeper CNN layer in the middle of the model. Without the inputs I can’t do a comparison. You can fine tune into a different model architecture. You’re right on not being able to retrain the model from scratch on half its data without that data but that’s likely pointless.

I’d be happy to be wrong about this but my understanding is that changing the architecture of the last few layers is feasible with fine-tuning but changing middle layers isn’t likely going to work very well without having the full original input set.

> likely pointless

It doesn’t take too much creativity to come up with ideas about why someone might want to do that:

- researchers who want to investigate how much the dataset can be reduced (and thus training cost) and what the accuracy penalty is

- someone who wants to for either religious or ethical reasons minimize the probability that the model was trained on pornography

- someone who’s curious about whether there’s significant redundancy in the existing input datasets

- someone who’s curious about whether there are a much smaller subset of images in the input dataset that can quickly help the first few CNN input layers converge before training the middle and output layers on the larger dataset.

Edit: I suspect the real reason they don’t want to share the input dataset is purely because a high-quality annotated dataset is a valuable commodity. While I don’t do ML work myself day-to-day, I do work with a team that does in a very niche field and I can only imagine how much effort they had to go through to get the annotated dataset that they’ve put together. Even just collecting the images for it involved many hours of drone flights in different locales around North America in varying weather and lighting.

Re: If you can't reproduce the model then it's not open-source

#96

Post by jncfhnb »

tonyarkles wrote: post #95
jncfhnb wrote: post #85

Earlier quoted context omitted.

> Maybe I think the model author should have used a deeper CNN layer in the middle of the model. Without the inputs I can’t do a comparison. You can fine tune into a different model architecture. You’re right on not being able to retrain the model from scratch on half its data without that data but that’s likely pointless.

I’d be happy to be wrong about this but my understanding is that changing the architecture of the last few layers is feasible with fine-tuning but changing middle layers isn’t likely going to work very well without having the full original input set. > likely pointless It doesn’t take too much creativity to come up with ideas about why someone might want to do that: - researchers who want to investigate how much the…

Original input set is irrelevant.

You will need some data of your own of course to fill in the blanks

Edit; however conversely, you can also splice out layers from one model into another original model. It’ll take some retraining, but this works!

Re: If you can't reproduce the model then it's not open-source

#98

Post by Trapais »

tqi wrote: post #38

"the project does not benefit from the OSS feedback loop" It's not like you can submit PRs to training data that fixes specific issues the way you can submit bug fixes, so I'm skeptical you would see much of a feedback loop. "it’s hard to verify that the model has no backdoors (eg sleeper agents)" Again given the size of the datasets and the opaque way training works, I am skeptical that anyone would be able tell if…

You can grep for bad words. What you can't do(unless hoops are jumped through) is to verify that weights came from the same dataset. You can set the same random seed and still get different results. Calculations are not that deterministic. (https://pytorch.org/docs/stable/notes/randomness.html#reprod...).

>I am overall skeptical that this is true in the case of LLMs

This skepticism seems reasonable. EleutherAI have documentation to reproduce training (https://github.com/EleutherAI/pythia#reproducing-training). So far I haven't seen it leading to anything. Lots of arxiv papers I've seen complain about time and budget constraint even regarding finetunes, forget pretraining.

Re: If you can't reproduce the model then it's not open-source

#99

Post by nullc »

jncfhnb wrote: post #87
anothernewdude wrote: post #81

Earlier quoted context omitted.

Yeah, that's why data scientists are out there editing the weights rather than cleaning up datasets and rerunning training with different settings.

If that was supposed to be clever it just sounds naive. There’s a ton of work going on fine tuning open source models

> There’s a ton of work going on fine tuning

... models provided in weights only form. (mostly!)

I believe the preferred form would be the whole kit and caboodle: the collection and filtering scripts, the data to the extent that it's non-public, the training routine, and the model weights... because sometimes you'll perform changes at any of those stages.

Post reply on HN