Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

If you can't reproduce the model then it's not open-source

twitter.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: If you can't reproduce the model then it's not open-source

#71

Post by wlesieutre »

jncfhnb wrote: post #70
wlesieutre wrote: post #68

Earlier quoted context omitted.

When you train an updated model on a new dataset do you really start by deleting all of the data that you collected for training the previous version?

Probably not. But if it’s the new data providing the advantage then you’re not exactly better off having the old data and the model vs. just having the model.

The idea would be that another group could fork it and continue adding to the dataset on their own.

As opposed to not being able to fork it at all because an "open source" model actually just means "you are allowed to use this particular release of our mystery box."

Re: If you can't reproduce the model then it's not open-source

#72

Post by nick238 »

tqi wrote: post #38

"the project does not benefit from the OSS feedback loop" It's not like you can submit PRs to training data that fixes specific issues the way you can submit bug fixes, so I'm skeptical you would see much of a feedback loop. "it’s hard to verify that the model has no backdoors (eg sleeper agents)" Again given the size of the datasets and the opaque way training works, I am skeptical that anyone would be able tell if…

The company policy/backdoors issues are possibly like the whole Getty Images debacle. If a company contracts with a provider or just uses a given model themselves, they may have no idea that it's taking from a ton of copyrighted work AND with enough of a trail where the infringed party could probably win a suit.

Backdoors I'd think of is if there are some sneaky words (maybe not even english) that all of a sudden causes it to emit NSFW outputs. Microsoft's short-lived @TayandYou comes to mind (but I don't think anyone's making that mistake again, where multiple users' sessions are pooled).

Re: If you can't reproduce the model then it's not open-source

#73

Post by [deleted] »

ssgodderidge wrote: post #2

> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.

[deleted]

Re: If you can't reproduce the model then it's not open-source

#74

Post by jncfhnb »

wlesieutre wrote: post #71
jncfhnb wrote: post #70

Earlier quoted context omitted.

Probably not. But if it’s the new data providing the advantage then you’re not exactly better off having the old data and the model vs. just having the model.

The idea would be that another group could fork it and continue adding to the dataset on their own. As opposed to not being able to fork it at all because an "open source" model actually just means "you are allowed to use this particular release of our mystery box."

You do not need the original dataset to train the model on an additional dataset

Maybe I misunderstood your original question. To be clear, the process of modifying a trained model does not require the presence of the original data. You said “deleted” which perhaps I misinterpreted. You’re not “instantiating a new model from scratch” when you modify it. You’re continuing to train it where it left off.

Re: If you can't reproduce the model then it's not open-source

#75

Post by tonyarkles »

jncfhnb wrote: post #64
ssgodderidge wrote: post #2

> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.

No it’s not. You have everything you need to modify the models to your own liking. You can explore how it works. This analogy is bad. Models are unlike code bases in this way.

> You have everything you need to modify the models to your own liking.

What if I wanted to train it using only half of its training set? If the inputs that were used to generate the set of released weights are not available I can’t do that. I have a set of weights and the model structure but without the training dataset I have no way of doing that.

To riff on the parent post, I have:

    Source + Compiler => Binaries
For the vast majority of open source models I have:

    [unavailable inputs] + Model Structure => Weights
They’re not exactly the same as the source code/binary scenario because I can still do this (which isn’t generally possible with binaries):

    Model Structure + Weights + [my own training data] => New Weights
Another way to look at it is that with source code I can modify the code and recompile it from scratch. Maybe I think the model author should have used a deeper CNN layer in the middle of the model. Without the inputs I can’t do a comparison.

Re: If you can't reproduce the model then it's not open-source

#76

Post by wlesieutre »

jncfhnb wrote: post #74
wlesieutre wrote: post #71

Earlier quoted context omitted.

The idea would be that another group could fork it and continue adding to the dataset on their own. As opposed to not being able to fork it at all because an "open source" model actually just means "you are allowed to use this particular release of our mystery box."

You do not need the original dataset to train the model on an additional dataset Maybe I misunderstood your original question. To be clear, the process of modifying a trained model does not require the presence of the original data. You said “deleted” which perhaps I misinterpreted. You’re not “instantiating a new model from scratch” when you modify it. You’re continuing to train it where it left off.

What if you want to start with a subset of the original data? Like you've trained a model, and then later said "You know, this new data we're adding is great, but maybe pulling all those comments from 4chan earlier was a mistake," wouldn't that require starting fresh with access to the actual data?

Re: If you can't reproduce the model then it's not open-source

#77

Post by tonyarkles »

jncfhnb wrote: post #62
nullc wrote: post #30

Earlier quoted context omitted.

The GPL describes the source as the "preferred form for modification".

The model weights ARE the preferred form for modification

Unless you want to try modifying the model structure, in which case the weights aren’t necessarily valid anymore and will need to be retrained.

Re: If you can't reproduce the model then it's not open-source

#78

Post by patmorgan23 »

CHY872 wrote: post #57
nullc wrote: post #30

Earlier quoted context omitted.

The GPL describes the source as the "preferred form for modification".

And, that's obviously fun, because with LLMs, you have the LLM itself which cost hundreds of thousands in compute to train, but given you have the weights it's eminently fine-tunable. So it's actually not really like Linux - rather it's closer to something like a car, where you had no hope of making it in the first place but now you have it, maybe you can modify it.

So in this case, the weights are the source code and the training material + compute time is like the software development process that went into creating the source code.

It would probably take well over a million dollars in engineering hours to recreate the postgres source code from scratch, just as it would take millions in compute to rebuild the weights.

Re: If you can't reproduce the model then it's not open-source

#79

Post by PeterisP »

jncfhnb wrote: post #62
nullc wrote: post #30

Earlier quoted context omitted.

The GPL describes the source as the "preferred form for modification".

The model weights ARE the preferred form for modification

As long-time a 'practitioner' of machine learning models I strongly disagree, the preferred form for model modification is by retraining the model with a tweak to the parameters or the training algorithm or the model structure or data selection or length of training.

You can get some effects by fine tuning, and in that case it may be preferable as it's cheaper, but in general if I want to have a different or better model, that involves retraining.

Re: If you can't reproduce the model then it's not open-source

#80

Post by anothernewdude »

jncfhnb wrote: post #64
ssgodderidge wrote: post #2

> Imagine if Linux published only a binary without the codebase. Or published the codebase without the compiler used to make the binary. This is where we are today. This was such a helpful way to frame the problem! Something felt off about the "open source models" out there; this highlights the problem incredibly well.

No it’s not. You have everything you need to modify the models to your own liking. You can explore how it works. This analogy is bad. Models are unlike code bases in this way.

You can do the same with binaries. Can modify those all you want.

Models are the compiler + makefiles. Dataset is the code.

Post reply on HN