Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

anyscale.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#31

Post by kaibee »

behnamoh wrote: post #29
redox99 wrote: post #27

Earlier quoted context omitted.

Potentially it could have an impact if it omits a high level description before writing the code, although obviously things like "Sure! Happy to help" do not help. In practice I haven't seen it make too much of a difference with GPT . The model can still use comments to express itself. For non coding tasks, adding "Think step by step" makes a huge difference (versus YOLOing a single word reply).

> although obviously things like "Sure! Happy to help" do not help. Yes you're right. I'm mostly concerned with the text that actually "computes" something before the actual code begins. Niceties like "sure! happy to help" don't compute anything. CoT indeed works. Now I've seem people take it to the extreme by having tree of thoughts, forest of thoughts, etc. but I'm not sure how much "reasoning" we can extract from…

> But then again, it's like trying to achieve perpetual motion in physics. One can't get more intelligence from a system than one puts in the system.

Not necessarily the same thing, as you're still putting in more processing power/checking more possible paths. Its kinda like simulated annealing, sure the system is dumb, but as long as checking if you have a correct answer is cheap, it still narrows down the search space a lot.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#32

Post by grandpayeti »

rising-sky wrote: post #30

> ~14 min. for 7B for 1 epoch on 3.5M tokens. ~26 min for 13B for 1 epoch. > At least 1xg5.16xlarge for head-node and 15xg5.4xlarge for worker nodes for both 7B and 13B For the uninitiated, anyone have an idea how much this would cost on AWS?

g5.16xlarge - $4.0960/hour

g5.4xlarge - $1.6240/hour

You're looking at about $30/hour to run this in us-east-1.

https://instances.vantage.sh/?selected=g5.16xlarge,g5.4xlarg...

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#33

Post by crooked-v »

redox99 wrote: post #20
behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Use a better model. airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code". https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

It's not useful for code, but you can see the difference of approach with NovelAI's homegrown Kayra model, which is set up to handle a mix of text completion and instruct functionality. It never includes extraneous prefix/suffix text and will smoothly follow instructions embedded in text without interrupting the text.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#34

Post by scv119 »

DebtDeflation wrote: post #22
perplexitywiz wrote: post #10

Earlier quoted context omitted.

https://twitter.com/Tim_Dettmers/status/1689375417189412864

I'm not sure to whom he is responding, since no one is claiming LoRA performs as well as traditional fine tuning. If you click through to the original Tweet he shared, it says "when you have a lot of data and limited compute go for LoRA, while with limited data and ample compute go for full finetuning" which I think is absolutely correct and few would disagree. As these models get bigger and bigger though, fewer and…

The tweet is referring to a paper that fine tunes Chinese dataset on english base model. I'm not surprised with LoRA's poor result in this setup.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#35

Post by charcircuit »

sillysaurusx wrote: post #28
ilaksh wrote: post #25

One challenge is that to get large enough custom datasets you either need a small army or a very strong existing model. Which means that you probably have to use OpenAI. And using OpenAI to generate training material for another model violates their terms. Has anyone taken them to court about this? Do we all just decide it's not fair and ignore it?

Why not ignore ToS? The worst that can happen is that you lose access.

The worst that can happen is you get brought into an expensive lawsuit.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#36

Post by zby »

SOLAR_FIELDS wrote: post #11
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

What is the general thought process on when it makes sense to use RAG vs fine tuning? How does segmenting fine tuning models make sense? Do I need a terraform LLM, a SQL LLM, and a python LLM, or can I just use a “code” LLM?

There is an article at the original site about that: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa...

Everybody new to this field thinks that he needs finetuning to teach the LLM of new facts. I made the same mistake initially, later I published a slightly ranty post on that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#37

Post by yousif_123123 »

DebtDeflation wrote: post #22
perplexitywiz wrote: post #10

Earlier quoted context omitted.

https://twitter.com/Tim_Dettmers/status/1689375417189412864

I'm not sure to whom he is responding, since no one is claiming LoRA performs as well as traditional fine tuning. If you click through to the original Tweet he shared, it says "when you have a lot of data and limited compute go for LoRA, while with limited data and ample compute go for full finetuning" which I think is absolutely correct and few would disagree. As these models get bigger and bigger though, fewer and…

I'm not sure less data should require full fine-tuning. If I had 5 pages of text, I don't see why I need to train billions of parameters that are already trained pretty well on general internet knowledge, and already know how to chat..

From a practical perspective, unless cost is really immaterial, I think most will end up starting with Lora, especially for 13b or 70b models.. you could do 10 fine-tuning runs for the cost of a few full fine-tunings.

But it's still all witchcraft to me to some degree, and I'd probably try full and Lora.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#38

Post by kouroshh »

bugglebeetle wrote: post #2

Glad to see the NER-like task performed the best, as I was just about to test something like this for comparison with a fine-tuned BERT model. Any idea about the training costs for this task?

Hey, I am one of the co-authors of the post. So the training data for ViGGO has about 5.1k rows which we trained with a block size of 512 (you can lower the block size if you want but we didn't do so because it was just easier to not change code :)). On 16xA10Gs for 7B it took ~15 min per epoch and on 13B it took ~25 min per epoch. So the on-demand cost per epoch is ~$7.2 for 7B and ~$12 for 13B. This is based on the time only spent on the training part and does not take into account the cluster startup time and shutdown time.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#39

Post by sandGorgon »

jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

this is brilliant. could you do a series about how to prepare custom data sets for finetuning. thats the part that a lot of other tutorials skip on. Especially for different goals - like safety, accuracy, etc.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#40

Post by rising-sky »

grandpayeti wrote: post #32
rising-sky wrote: post #30

> ~14 min. for 7B for 1 epoch on 3.5M tokens. ~26 min for 13B for 1 epoch. > At least 1xg5.16xlarge for head-node and 15xg5.4xlarge for worker nodes for both 7B and 13B For the uninitiated, anyone have an idea how much this would cost on AWS?

g5.16xlarge - $4.0960/hour g5.4xlarge - $1.6240/hour You're looking at about $30/hour to run this in us-east-1. https://instances.vantage.sh/?selected=g5.16xlarge,g5.4xlarg...

thanks
Post reply on HN