101–110 of 117 posts
Earlier quoted context omitted.
You can do the same with binaries. Can modify those all you want. Models are the compiler + makefiles. Dataset is the code.
No, it’s not the same. Yes, you can technically modify binaries, but it’s not at all the preferred way to modify the program.
Earlier quoted context omitted.
If that was supposed to be clever it just sounds naive. There’s a ton of work going on fine tuning open source models
> There’s a ton of work going on fine tuning ... models provided in weights only form. (mostly!) I believe the preferred form would be the whole kit and caboodle: the collection and filtering scripts, the data to the extent that it's non-public, the training routine, and the model weights... because sometimes you'll perform changes at any of those stages.
Earlier quoted context omitted.
> There’s a ton of work going on fine tuning ... models provided in weights only form. (mostly!) I believe the preferred form would be the whole kit and caboodle: the collection and filtering scripts, the data to the extent that it's non-public, the training routine, and the model weights... because sometimes you'll perform changes at any of those stages.
Do you actually do this for a living? Do you have experience doing this and have credibility talking about what’s preferred? I do.
Are there any true open-source LLM models, where all the training data is publicly-available (with a compatible license) and the training software can reproduce bit-identical models? Is training nondeterministic? I know LLM outputs are purposely nondeterministic.
Mamba has a version, trained on publicly available SlimPajama. RedPajama-INCITE was trained on non-slimmed version of the dataset(it's only one dataset).
I'm not sure if training scripts are available.
Pythia definitely has scripts. However it was trained on the pile, so you have to find books3 on your own.
Also I believe LLM360 is an explicit attempt to do it with llama.
>Is training nondeterministic?
Correct. Torch documentation has a section on reproducibility of a training.
Earlier quoted context omitted.
Somewhat unrelated, but here is a thought experiment... If a human knows a song "by heart" (imperfectly), it is not considered copyright infringement. If a LLM knows a song as part of its training data, then it is copyright infringement. But what if you developed a model with no prepared training data and forced it to learn from it's own sensory inputs. Instead of shoveling it bits, you played it this particular song…
If a person plays a song similarly enough, then it is copyright infringment! Mere knowledge is irrelevant, it is the producing of copies (and also a few related actions) which is prohibited by copyright.
Mere knowledge is irrelevant only because we don't (yet) have a mechanism to pry open one's brains and inspect the copying of songs within different parts of one's brain. Otherwise, mechanistically, besides one using silicon and other using wetware, they're pretty much doing the same thing.
Earlier quoted context omitted.
Do you actually do this for a living? Do you have experience doing this and have credibility talking about what’s preferred? I do.
OK. Where is your reproduction of Pythia trained from scratch? Or MPT? Or Amber? Shall we play a game where you give paper regarding pretraining (and we are not taling about puny models based on wikitext2) I give you a paper based around finetuning and we'll see who run out of papers first?
I see you’re someone else, so I’ll ask you too. Do you actually have any experience doing this? Have you ever fine tuned models or tried to change architecture or put a piece of one model into another?
Earlier quoted context omitted.
But one of the four freedoms is being able to modify/tweek things, including the model. If all you have is the model weights, then you can't easily tweak the model. The model weights is hardly the preferred form for making changes to update the model. The equivalent would be someone which gives you only the binary to Libreoffice. That's perfectly fine for editing documents and spreadsheets, but suppose you want to fi…
If all you have are the model weights, you can very easily tweak the model. How else are all these "decensored" Llama2 showing up on Hugging Face? There's a lot of value in a trained LLM model itself and it's 100% a type of openness to release these trained models. What you can't easily do is retrain from scratch using a heavily modified architecture or different training data preconditioning. So yes, it is valuable…
And yet, the people who insist on having source code so they can edit the program and recompile it have said that for programs, having just the binary isn't good enough.
Earlier quoted context omitted.
But one of the four freedoms is being able to modify/tweek things, including the model. If all you have is the model weights, then you can't easily tweak the model. The model weights is hardly the preferred form for making changes to update the model. The equivalent would be someone which gives you only the binary to Libreoffice. That's perfectly fine for editing documents and spreadsheets, but suppose you want to fi…
Next step will be to ask for GPU time. Because even with data, model code and training framework you may have no resources to train. "The equivalent would be" someone gives you the code, but no access to mainframe which is required to compile. Which would make it not open source(?) There are other variations, like original compiler was lost, current compilers aren't backward compatible. Does that make old open source…
PS: If I ever make my models open I can't open the data anyway. License on images directly prohibits publishing them.