Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

anyscale.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#21

Post by kouroshh »

behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Llama-2-chat models have been overly fine-tuned to be like this. You can give a few-shot prompting a try, but they still don't gurantee a desired output. The best way to guarantee is to fine-tune on small (~1k) data points and go from there.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#22

Post by DebtDeflation »

perplexitywiz wrote: post #10
yousif_123123 wrote: post #9

It's weird that Lora and training with quantization is not being taken more seriously. It's way cheaper, takes less time, and a lot of evidence shows it's pretty good. I don't think it should be something brushed on the side to be tried out later..

https://twitter.com/Tim_Dettmers/status/1689375417189412864

I'm not sure to whom he is responding, since no one is claiming LoRA performs as well as traditional fine tuning. If you click through to the original Tweet he shared, it says "when you have a lot of data and limited compute go for LoRA, while with limited data and ample compute go for full finetuning" which I think is absolutely correct and few would disagree. As these models get bigger and bigger though, fewer and fewer people are going to have the "ample compute" required for full fine tuning.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#23

Post by kouroshh »

nacs wrote: post #12
SubiculumCode wrote: post #8

Earlier quoted context omitted.

one gpu? feasible with one 3060?

Absolutely. For QLORA / 4bit / GPTQ finetuning, you can train a 7B easily on an RTX 3060 (12GB VRAM). If you have a 24GB VRAM GPU like a RTX 3090/4090, you can Qlora finetune a 13B or even a 30B model (in a few hours).

Would be good to see a rigorous analysis of these PEFT methods on quality. There still seems to be a debate on whether these methods sacrifice quality or not.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#24

Post by behnamoh »

redox99 wrote: post #20
behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Use a better model. airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code". https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

Thanks, I'll give this a try.

I wonder if LLMs will have less reasoning power if they simply return the output. AFAIK, they think by writing their thoughts. So forcing an LLM to just return the goddamn code might limit its reasoning skills, leading to poor code. Is that true?

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#25

Post by ilaksh »

One challenge is that to get large enough custom datasets you either need a small army or a very strong existing model. Which means that you probably have to use OpenAI. And using OpenAI to generate training material for another model violates their terms.

Has anyone taken them to court about this? Do we all just decide it's not fair and ignore it?

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#26

Post by bugglebeetle »

spdustin wrote: post #4

Seeing NER examples pop up more frequently now, and wondering why folks don’t use spacy for those sorts of tasks.

Spacy doesn’t work well for multilingual training data and I’ve found it barfs in more and somehow even odder ways than stuff in transformers.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#27

Post by redox99 »

behnamoh wrote: post #24
redox99 wrote: post #20

Earlier quoted context omitted.

Use a better model. airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code". https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

Thanks, I'll give this a try. I wonder if LLMs will have less reasoning power if they simply return the output. AFAIK, they think by writing their thoughts. So forcing an LLM to just return the goddamn code might limit its reasoning skills, leading to poor code. Is that true?

Potentially it could have an impact if it omits a high level description before writing the code, although obviously things like "Sure! Happy to help" do not help.

In practice I haven't seen it make too much of a difference with GPT. The model can still use comments to express itself.

For non coding tasks, adding "Think step by step" makes a huge difference (versus YOLOing a single word reply).

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#28

Post by sillysaurusx »

ilaksh wrote: post #25

One challenge is that to get large enough custom datasets you either need a small army or a very strong existing model. Which means that you probably have to use OpenAI. And using OpenAI to generate training material for another model violates their terms. Has anyone taken them to court about this? Do we all just decide it's not fair and ignore it?

Why not ignore ToS? The worst that can happen is that you lose access.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#29

Post by behnamoh »

redox99 wrote: post #27
behnamoh wrote: post #24

Earlier quoted context omitted.

Thanks, I'll give this a try. I wonder if LLMs will have less reasoning power if they simply return the output. AFAIK, they think by writing their thoughts. So forcing an LLM to just return the goddamn code might limit its reasoning skills, leading to poor code. Is that true?

Potentially it could have an impact if it omits a high level description before writing the code, although obviously things like "Sure! Happy to help" do not help. In practice I haven't seen it make too much of a difference with GPT . The model can still use comments to express itself. For non coding tasks, adding "Think step by step" makes a huge difference (versus YOLOing a single word reply).

> although obviously things like "Sure! Happy to help" do not help.

Yes you're right. I'm mostly concerned with the text that actually "computes" something before the actual code begins. Niceties like "sure! happy to help" don't compute anything.

CoT indeed works. Now I've seem people take it to the extreme by having tree of thoughts, forest of thoughts, etc. but I'm not sure how much "reasoning" we can extract from a model that is obviously limited in terms of knowledge and intelligence. CoT already gets us to 80% of the way. With some tweaks it can get even better.

I've also seen simulation methods where GPT "agents" talk to each other to form better ideas about a subject. But then again, it's like trying to achieve perpetual motion in physics. One can't get more intelligence from a system than one puts in the system.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#30

Post by rising-sky »

> ~14 min. for 7B for 1 epoch on 3.5M tokens. ~26 min for 13B for 1 epoch.

> At least 1xg5.16xlarge for head-node and 15xg5.4xlarge for worker nodes for both 7B and 13B

For the uninitiated, anyone have an idea how much this would cost on AWS?

Post reply on HN