Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

github.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#11

Post by brucethemoose2 »

flangola7 wrote: post #8
minosu wrote: post #2

This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages. gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a sing…

I'm not sure what MIT licensed code is supposed to do for you. Are you going to cite every repository ingested?

Technically you could do that in a big text file...

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#13

Post by brucethemoose2 »

mihir_ch wrote: post #9

In the future could we see a models fine-tuned for specializing in every language? Or would a general model outperform?

Finetunes are historically better than the base model, thats the whole idea of a finetune... but I think you are really asking if finetunes of public models will be as good as huge SOTA proprietary models and their paid "finetuning" services.

Thats hard to say.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#14

Post by minosu »

the_tli wrote: post #4

Thanks for sharing. Why is the training dataset that contains instructions and output wrapped by another enclosing prompt ( https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622... ) Why does this even work when during inference this wrapping prompt is absent? Wouldnt the model then work best against a inference prompt that follows the wrapping prompt structure, however the desired outcome is to have a model t…

The wrapping prompt is also used during inference. (https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622...) Prompting like this is useful for instruct-finetunes, and similar prompts are used by other projects like stanford-alpaca.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#15

Post by the_tli »

minosu wrote: post #14
the_tli wrote: post #4

Thanks for sharing. Why is the training dataset that contains instructions and output wrapped by another enclosing prompt ( https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622... ) Why does this even work when during inference this wrapping prompt is absent? Wouldnt the model then work best against a inference prompt that follows the wrapping prompt structure, however the desired outcome is to have a model t…

The wrapping prompt is also used during inference. ( https://github.com/minosvasilias/godot-dodo/blob/f62b90a4622... ) Prompting like this is useful for instruct-finetunes, and similar prompts are used by other projects like stanford-alpaca.

Thanks for the clarification, makes sense now!

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#16

Post by brucethemoose2 »

AaronFriel wrote: post #10

The performance report doesn't describe the loss approached by each of these fine tunings, but I wonder if the number of tokens in the instruction dataset was just not nearly long enough to produce high quality output. I can't think of any other reason the 13B parameter model would perform worse than the 7B model. Would love to see a deep dive into the fine tuning and more details - by epoch if possible - on the outp…

I have seen this same phenomenon mentioned on huggingface: a finetuned large model being worse than its smaller variant.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#17

Post by minosu »

omoikane wrote: post #6

On this page: https://github.com/minosvasilias/godot-dodo/tree/main/models It seems that some of the GPT syntax errors appear to be because the models were trained for Godot 3, but the tests were conducted against Godot 4, hence the error messages like "KinematicBody2D does not exist in 4.x (should be CharacterBody2D)".

Yes, the changes introduced by Godot 4 were a prime motivator for this project.

However, it is not quite as clear cut as OpenAI's models simply being trained on Godot 3.x projects only. Not only do they sometimes produce valid 4.x syntax (gpt-4 more often than 3.5-turbo), indicating there were at least some 4.x projects in the training data, they also hallucinate other invalid syntax, such as Python-specific functionality, or simply non-existent methods.

I do think evaluating against Godot 3.x would increase their scores somewhat, but i have not had time to do so yet.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#20

Post by sp332 »

throwme_123 wrote: post #19

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

A100's cost about $4/hour/GPU to rent. So the total cost depends on the amount of time, but 8x for 24 hours would cost $768.
Post reply on HN