Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

anyscale.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#11

Post by SOLAR_FIELDS »

jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

What is the general thought process on when it makes sense to use RAG vs fine tuning?

How does segmenting fine tuning models make sense? Do I need a terraform LLM, a SQL LLM, and a python LLM, or can I just use a “code” LLM?

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#12

Post by nacs »

SubiculumCode wrote: post #8
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

one gpu? feasible with one 3060?

Absolutely. For QLORA / 4bit / GPTQ finetuning, you can train a 7B easily on an RTX 3060 (12GB VRAM).

If you have a 24GB VRAM GPU like a RTX 3090/4090, you can Qlora finetune a 13B or even a 30B model (in a few hours).

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#13

Post by DebtDeflation »

SOLAR_FIELDS wrote: post #11
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

What is the general thought process on when it makes sense to use RAG vs fine tuning? How does segmenting fine tuning models make sense? Do I need a terraform LLM, a SQL LLM, and a python LLM, or can I just use a “code” LLM?

Fine tuning for training the model to perform a new task, RAG for adding knowledge.

In your example, you would fine tune the model to train it to code in a language it hasn't seen before, RAG will not really help with that.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#14

Post by binarymax »

bugglebeetle wrote: post #2

Glad to see the NER-like task performed the best, as I was just about to test something like this for comparison with a fine-tuned BERT model. Any idea about the training costs for this task?

Great question. I wish they said how long the 10 epochs took, so we could figure out the cost (or better, just posted the time and cost together):

"For the 7B and 13B models, we used 16xA10Gs, and for the 70B model, we used 32xA10Gs (across 4x g5.48xlarge instances). When using Ray, there's no need to secure A100s to perform full-parameter fine-tuning on these models! The process is simply repeated for each task. Figures below show an example run based on a context length of 512, with a total of 3.7M effective tokens per epoch on GSM8k dataset.

We ran the training for a maximum of 10 epochs and selected the best checkpoint according to the minimum perplexity score on the validation set."

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#15

Post by jawerty »

SOLAR_FIELDS wrote: post #11
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

What is the general thought process on when it makes sense to use RAG vs fine tuning? How does segmenting fine tuning models make sense? Do I need a terraform LLM, a SQL LLM, and a python LLM, or can I just use a “code” LLM?

I have an RAG video (my "make a ChatGPT with podcasts" video) you might be interested in. Semantic search is increddible and you might be surprised how good a Q/A solution is by just extracting passages that answer the question.

Overall it depends on whether or not you can turn your data into a fine-tuning data and if you can find a low parameter (enough) model that can use your found contexts as input to host yourself of use inference endpoints. Hosting an LLM is actually not easy and I'm finding in the field working an information retrieval business OpenAI isn't terrible compared to costs of having a GPUs for your users across the world.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#16

Post by jawerty »

nacs wrote: post #12
SubiculumCode wrote: post #8

Earlier quoted context omitted.

one gpu? feasible with one 3060?

Absolutely. For QLORA / 4bit / GPTQ finetuning, you can train a 7B easily on an RTX 3060 (12GB VRAM). If you have a 24GB VRAM GPU like a RTX 3090/4090, you can Qlora finetune a 13B or even a 30B model (in a few hours).

+1 this

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#17

Post by behnamoh »

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”).

That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Does anyone know if there's any prompt technique to fix this problem?

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#18

Post by binarymax »

spdustin wrote: post #4

Seeing NER examples pop up more frequently now, and wondering why folks don’t use spacy for those sorts of tasks.

My line of thinking is using the more expensive model to label data, then use a teacher/student methodology to train the smaller model (SpaCy or BERT) for cost & speed.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#19

Post by kouroshh »

binarymax wrote: post #14
bugglebeetle wrote: post #2

Glad to see the NER-like task performed the best, as I was just about to test something like this for comparison with a fine-tuned BERT model. Any idea about the training costs for this task?

Great question. I wish they said how long the 10 epochs took, so we could figure out the cost (or better, just posted the time and cost together): "For the 7B and 13B models, we used 16xA10Gs, and for the 70B model, we used 32xA10Gs (across 4x g5.48xlarge instances). When using Ray, there's no need to secure A100s to perform full-parameter fine-tuning on these models! The process is simply repeated for each task. Fig…

Training times for GSM8k are mentioned here: https://github.com/ray-project/ray/tree/master/doc/source/te...

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#20

Post by redox99 »

behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Use a better model.

airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code".

https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

Post reply on HN