Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
1–10 of 29 posts
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#2This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages.
gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a single one of these languages, using human-created code scraped from MIT-licensed GitHub repositories, with existing GPT models generating instructions for each code snippet.
This differs from the dataset generation approach used by projects such as stanford-alpaca or gpt4all, in that the output values of the training set remain high quality, human data, while following the same instruction-following behavior. This will likely prove more effective the more obscure the language. In this case, GDScript was used, which is the scripting language for the popular open-source game-engine Godot. The same approach however can be applied to any other language.
Performance is promising, with the 7 billion parameter finetune outperforming GPT models in producing syntax that compiles on first try, while being somewhat less capable at following complex instructions.
A comprehensive evaluation comparing all models can be found here: https://github.com/minosvasilias/godot-dodo/tree/main/models
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#3This is fabulous.
Just want to add that there are efforts to impove training speed, like this: https://github.com/Lightning-AI/lit-llama/issues/62
So the practical cost/dataset size for language finetunes is bound to get better rapidly.
EDIT: And there is also this for JAX finetuning. https://github.com/young-geng/EasyLM/blob/main/docs/llama.md
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#4Thanks for sharing. Why is the training dataset that contains instructions and output wrapped by another enclosing prompt (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...)
Why does this even work when during inference this wrapping prompt is absent? Wouldnt the model then work best against a inference prompt that follows the wrapping prompt structure, however the desired outcome is to have a model that just works without the wrapping prompt?
Edit: see reply from OP, the wrapping prompt is used for inference as well, so misunderstanding on my part
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#5This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages. gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a sing…
This sounds like one of those bootstrapping liftoff things. Generating labels had been a big bottleneck, but if we can just find examples and then label them automatically, this could accelerate all sorts of applications.
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#6On this page:
https://github.com/minosvasilias/godot-dodo/tree/main/models
It seems that some of the GPT syntax errors appear to be because the models were trained for Godot 3, but the tests were conducted against Godot 4, hence the error messages like "KinematicBody2D does not exist in 4.x (should be CharacterBody2D)".
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#7This is nice work, and it is great to see the effort taken to show the pipeline so this will work for others.
One further extension might be to fine-tune to specifically encourage behavior for a client like godot-copilot. I bet you could teach it to obey your particular prompt structure (eg, matching indentation, or inserting code at the right spot without adding it elsewhere). That would really complete the story to make this very usable by everyday people who dont know/care about LLM internals and fine tunings.
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#8This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages. gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a sing…
I'm not sure what MIT licensed code is supposed to do for you. Are you going to cite every repository ingested?
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#9In the future could we see a models fine-tuned for specializing in every language? Or would a general model outperform?
Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs
#10The performance report doesn't describe the loss approached by each of these fine tunings, but I wonder if the number of tokens in the instruction dataset was just not nearly long enough to produce high quality output.
I can't think of any other reason the 13B parameter model would perform worse than the 7B model. Would love to see a deep dive into the fine tuning and more details - by epoch if possible - on the output.