I'm curious during training if all the weights change compared LLaMA?
Yeah the weights did change.
CodeAlpaca – Instruction following code generation model
31–40 of 50 posts
Re: CodeAlpaca – Instruction following code generation model
#32The results are pretty good; I wish they'd just publish the models so we can run the inference locally (not too many people have access to 8xA100 to train themselves, though I appreciate including the training data and instructions too).
Given how many people are asking for the weights, I'm working to make them available now. So you'll be able to run them locally as well.
Re: CodeAlpaca – Instruction following code generation model
#33Anyone know of a 65B instruct model I can run on llama.cpp? Or if it’s possible to train one on a 64gb Mac Studio? Haven’t come across either yet.
I'm training a 65B model right now, also I believe you can use lora-alpaca to train on this data on a much smaller machine.
Re: CodeAlpaca – Instruction following code generation model
#34The results are pretty good; I wish they'd just publish the models so we can run the inference locally (not too many people have access to 8xA100 to train themselves, though I appreciate including the training data and instructions too).
Llama may not be licensed for people to share since you need to apply to get one from Facebook for non commercial use. I think it's more of a license issue Hopefully similar work can be done with LoRA so the fine-tuning is not as expensive
Yeah the license issue is the only reason why I didn't release the weights yet.
Re: CodeAlpaca – Instruction following code generation model
#35If someone were to add noise to the llama weights and then retrain a little, would anyone be able to tell? Could that org then pass it off as their own training, MIT licensed for the good of humanity?
Re: CodeAlpaca – Instruction following code generation model
#36> The code runs on a 8xA100 80GB, but can also run on 8xA10040GB or 4xA100 with lower batch size and gradient accumulation steps. To get the GPUs, I suggest using Lambda Labs, best pricing for the best hardware. I wonder how much it was total in $ for the fine-tuning. Also, does anyone have some sort of table/formula that relates MB/GB of training data to $ for fine-tuning?
All included it costs under 70$ for the 13B model. Training 65B now so will report what that will cost.
Please do! Also please include how you’re calculating the costs.
Re: CodeAlpaca – Instruction following code generation model
#37> The code runs on a 8xA100 80GB, but can also run on 8xA10040GB or 4xA100 with lower batch size and gradient accumulation steps. To get the GPUs, I suggest using Lambda Labs, best pricing for the best hardware. I wonder how much it was total in $ for the fine-tuning. Also, does anyone have some sort of table/formula that relates MB/GB of training data to $ for fine-tuning?
All included it costs under 70$ for the 13B model. Training 65B now so will report what that will cost.
For the 65B fine tune, did you add another A100 node? Or just drop batch size?
Any chance you’re up to sharing the training parameters?
Re: CodeAlpaca – Instruction following code generation model
#38Earlier quoted context omitted.
All included it costs under 70$ for the 13B model. Training 65B now so will report what that will cost.
For the 65B fine tune, did you add another A100 node? Or just drop batch size? Any chance you’re up to sharing the training parameters?
Dropping the batch size
Re: CodeAlpaca – Instruction following code generation model
#39If someone were to add noise to the llama weights and then retrain a little, would anyone be able to tell? Could that org then pass it off as their own training, MIT licensed for the good of humanity?
By looking at the weights? Yes, it would be very easy to tell.
Re: CodeAlpaca – Instruction following code generation model
#40Didn't explore much, but it seems alpaca-lora has better results for coding tasks. One example I've used was: "Implement quicksort in python.". This is the result with Code alpaca: def quicksort(arr): if len(arr) pivot: right_index -= 1 if left_index Alpaca lora result: def quicksort(arr): if len(arr) pivot] return quicksort(left) + [pivot] + quicksort(right) Shorter and much cleaner, not to mention it works (code al…
The Alpaca lora result is incorrect since, for example, if all elements in array are the same the result in just an array with one element.