Here is a thread to tweak the parameters which the model seems very sensitive to: https://github.com/ggerganov/llama.cpp/issues/129
Could the model itself be used to tweak it’s own parameters iteratively?
LLaMa running at 5 tokens/second on a Pixel 6
51–60 of 79 posts
Re: LLaMa running at 5 tokens/second on a Pixel 6
#52would be even cooler if it employed the accelerator!
(unless this ggml library is doing that under the hood)
i assume it has unified memory, but maybe not little numbers...
Re: LLaMa running at 5 tokens/second on a Pixel 6
#53Earlier quoted context omitted.
That's why Alpaca is so exciting: it instruction-tunes LLaMA to the point that even the tiny 7B model (the one that fits on a phone) produces useful output: https://simonwillison.net/2023/Mar/13/alpaca/
But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out
If they don't release the model, recreating it doesn't look too hard. $100 worth of compute time to run the fine-tuning, and the training data they used is here: https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpac...
That would have the same licensing problems that they have though: that alpaca_data.json file was created using GPT3. But creating a "clean" training set of 52,000 examples doesn't feel impossible to me for the right group.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#54Re: LLaMa running at 5 tokens/second on a Pixel 6
#55Earlier quoted context omitted.
But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out
If they don't release the model, recreating it doesn't look too hard. $100 worth of compute time to run the fine-tuning, and the training data they used is here: https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpac... That would have the same licensing problems that they have though: that alpaca_data.json file was created using GPT3. But creating a "clean" training set of 52,000 examples doesn't feel impossib…
You're only bound by the terms of OpenAI's agreement if you agreed to the terms of use. If a third party obtained the data without signing an agreement with OpenAI (eg. by just downloading it from that repo) they are under no obligation to refrain from using it to compete with OpenAI. It is fair-use by the same argument OpenAI itself uses to train its own models on publicly available data.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#56Earlier quoted context omitted.
But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
You don't need to find a new data source, you just need to find an unencumbered third party. You can use the that data publicly provided in the git repo as long as you haven't signed an agreement with OpenAI yourself.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#57Earlier quoted context omitted.
But they won’t give us the model… so it’s ultimately meaningless because they’ll just sell out
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
What will OpenAI do, sue? Okay but now it's out there.
Re: LLaMa running at 5 tokens/second on a Pixel 6
#58Earlier quoted context omitted.
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
What will OpenAI do, sue? Okay but now it's out there.
Also how could that even be under protection. As if they haven’t been scraping copyrighted materials and sites with end user agreements to train the model in the first place
Re: LLaMa running at 5 tokens/second on a Pixel 6
#59It is not really llama, it is llama quantized to 4bit. Not even the quality of original 7B. I could also quantize it to 1 bit and claim it runs on my RPI3.
The quantization to four hits doesn’t have that much effect on the output. 1 bit might not either, but someone would need to do some testing before making the claim that “1 bit … runs on my RPI3” because “runs” is a bit overloaded to mean “runs and produces sensible output.” I think you’re missing that runs here has that overloading.
It should also be mentioned that it isn’t really that each weight is a 4 bit float, but rather that they’re basically clustering floats into 2^4 clusters and then grabbing from a lookup table the float associated to a 4 bit value as needed. So as long as the weights roughly fall into 16 clusters you’ll get identical results
Re: LLaMa running at 5 tokens/second on a Pixel 6
#60Earlier quoted context omitted.
My understanding is they legally can't. It was trained used OpenAI, which doesn't allow using their output to train new models. Someone would need to find another data source to fine tune llama.
What will OpenAI do, sue? Okay but now it's out there.
I believe the work was done by Stanford, so OpenAI could revoke Stanford's access to their API. That would inhibit Stanford's ability to do new research with this system.