Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

github.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#21

Post by lsaferite »

throwme_123 wrote: post #19

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

They called out the costs incurred:

  $30 - dataset generation (OpenAI GPT 3.5-turbo)
  $24 - llama 7b fine-tuning (8x A100 80GB instance costs)
  $84 - llama 13b fine-tuning (8x A100 80GB instance costs)

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#22

Post by brucethemoose2 »

throwme_123 wrote: post #19

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model

What OP did is more intense, but Lora/Adapter still can give excellent results.

Pure AI cards for mere mortals aren't really a thing yet.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#23

Post by _flux »

flangola7 wrote: post #8
minosu wrote: post #2

This repository presents finetuned LLaMA models that try to address the limited ability of existing language models when it comes to generating code for less popular programming languages. gpt-3.5-turbo and gpt-4 have proven to be excellent coders, but fall off sharply when asked to generate code for languages other than Python/Javascript etc. The godot-dodo approach to address this: Finetune smaller models on a sing…

I'm not sure what MIT licensed code is supposed to do for you. Are you going to cite every repository ingested?

I suppose for the model indeed you should do that?

But then maybe not for the actual predictions made by the model, as the MIT license says:

> The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Arguably e.g. a single function is not a substantial portion of a multi-file project—and, usually, even that function itself is not going to be a verbatim copy but adjusted to your use case regarding variable names etc.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#25

Post by rvcdbn »

sharepower123 wrote: post #24

I think that many people could be interested in sharing cost if they can obtain a LLaMA based finetuned model better than GPT4 in their preferred language. So there is an opportunity for someone creating a startup just for that.

There have to be easier ways to share costs of running the same model, no?

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#26

Post by theolivenbaum »

brucethemoose2 wrote: post #22
throwme_123 wrote: post #19

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model What OP did is more intense, but Lora/Adapter still can give excellent results. Pure AI cards for mere mortals aren't really a thing yet.

Anyone tried fine tunning on CPU already? I expect it to be much slower, but is it even practical?

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#27

Post by brucethemoose2 »

theolivenbaum wrote: post #26
brucethemoose2 wrote: post #22

Earlier quoted context omitted.

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model What OP did is more intense, but Lora/Adapter still can give excellent results. Pure AI cards for mere mortals aren't really a thing yet.

Anyone tried fine tunning on CPU already? I expect it to be much slower, but is it even practical?

Oh no not a chance, finetuning is really compute intense.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#28

Post by akiselev »

sp332 wrote: post #20
throwme_123 wrote: post #19

> For finetuning godot_dodo_4x_60k_llama_13b, eight A100 80GB GPUs were used. $300k of hardware! Guess it answers my previous comment from Hetzner server https://news.ycombinator.com/item?id=35662925

A100's cost about $4/hour/GPU to rent. So the total cost depends on the amount of time, but 8x for 24 hours would cost $768.

8x A100 80GB are $12/hour or so on Lambda Cloud. If you get lucky enough to snag capacity.

Re: Godot-dodo – Finetuning LLaMA on single-language comment:code data pairs

#29

Post by brucethemoose2 »

theolivenbaum wrote: post #26
brucethemoose2 wrote: post #22

Earlier quoted context omitted.

You can customize a model with a "cheap" 3090 (or maybe a 7900 XTX), see here: https://github.com/Lightning-AI/lit-llama#finetune-the-model What OP did is more intense, but Lora/Adapter still can give excellent results. Pure AI cards for mere mortals aren't really a thing yet.

Anyone tried fine tunning on CPU already? I expect it to be much slower, but is it even practical?

Also I bet the pytorch training code is written with CUDA semantics. Maybe a JAX version would work without messing with the code.
Post reply on HN