Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

anyscale.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#41

Post by sandGorgon »

zby wrote: post #36
SOLAR_FIELDS wrote: post #11

Earlier quoted context omitted.

What is the general thought process on when it makes sense to use RAG vs fine tuning? How does segmenting fine tuning models make sense? Do I need a terraform LLM, a SQL LLM, and a python LLM, or can I just use a “code” LLM?

There is an article at the original site about that: https://www.anyscale.com/blog/fine-tuning-is-for-form-not-fa... Everybody new to this field thinks that he needs finetuning to teach the LLM of new facts. I made the same mistake initially, later I published a slightly ranty post on that: https://zzbbyy.substack.com/p/why-you-need-rag-not-finetunin...

Quick question - Gorilla paper talks about finetuning for RAG. Do you see this in practice ? can you do finetuning that specifically affects RAG ?

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#42

Post by mikeravkine »

redox99 wrote: post #20
behnamoh wrote: post #17

> Additionally, while this wasn’t an issue for GPT, the Llama chat models would often output hundreds of miscellaneous tokens that were unnecessary for the task, further slowing down their inference time (e.g. “Sure! Happy to help…”). That's the problem I've been facing with Llama 2 as well. It's almost impossible to have it just output the desired text. It will always add something before and after its response. Doe…

Use a better model. airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code". https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

The model card also has prompt formats for context aware document Q/A and multi-CoT, using those correctly improves performance at such tasks significantly.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#43

Post by behnamoh »

kaibee wrote: post #31
behnamoh wrote: post #29

Earlier quoted context omitted.

> although obviously things like "Sure! Happy to help" do not help. Yes you're right. I'm mostly concerned with the text that actually "computes" something before the actual code begins. Niceties like "sure! happy to help" don't compute anything. CoT indeed works. Now I've seem people take it to the extreme by having tree of thoughts, forest of thoughts, etc. but I'm not sure how much "reasoning" we can extract from…

> But then again, it's like trying to achieve perpetual motion in physics. One can't get more intelligence from a system than one puts in the system. Not necessarily the same thing, as you're still putting in more processing power/checking more possible paths. Its kinda like simulated annealing, sure the system is dumb, but as long as checking if you have a correct answer is cheap, it still narrows down the search sp…

> Its kinda like simulated annealing.

Yeah I get that. We assume there's X amount of intelligence in the LLM and try different paths to tap on that potential. The more paths are simulated, the closer we get to the LLM's intelligence asymptote. But then that's it—we can't go any further.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#44

Post by bugglebeetle »

kouroshh wrote: post #38
bugglebeetle wrote: post #2

Glad to see the NER-like task performed the best, as I was just about to test something like this for comparison with a fine-tuned BERT model. Any idea about the training costs for this task?

Hey, I am one of the co-authors of the post. So the training data for ViGGO has about 5.1k rows which we trained with a block size of 512 (you can lower the block size if you want but we didn't do so because it was just easier to not change code :)). On 16xA10Gs for 7B it took ~15 min per epoch and on 13B it took ~25 min per epoch. So the on-demand cost per epoch is ~$7.2 for 7B and ~$12 for 13B. This is based on the…

Great! Thank you!

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#45

Post by bugglebeetle »

ilaksh wrote: post #25

One challenge is that to get large enough custom datasets you either need a small army or a very strong existing model. Which means that you probably have to use OpenAI. And using OpenAI to generate training material for another model violates their terms. Has anyone taken them to court about this? Do we all just decide it's not fair and ignore it?

This is not true for all tasks. For many NLP tasks, you just need to reformat existing data to match the LLM format.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#46

Post by jawerty »

sandGorgon wrote: post #39
jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

this is brilliant. could you do a series about how to prepare custom data sets for finetuning. thats the part that a lot of other tutorials skip on. Especially for different goals - like safety, accuracy, etc.

Of course I have a few where I web scrape and build a dataset for myself with prefix tokens. I can break that down more on a specific stream about it.

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#47

Post by dontupvoteme »

behnamoh wrote: post #24
redox99 wrote: post #20

Earlier quoted context omitted.

Use a better model. airoboros supports the PLAINFORMAT token "to avoid backticks, explanations, etc. and just print the code". https://huggingface.co/TheBloke/airoboros-l2-70B-GPT4-2.0-GG...

Thanks, I'll give this a try. I wonder if LLMs will have less reasoning power if they simply return the output. AFAIK, they think by writing their thoughts. So forcing an LLM to just return the goddamn code might limit its reasoning skills, leading to poor code. Is that true?

You can also just parse the text for all valid code blocks and combine them. I have a script which automatically check the clipboard for this

There's no reason to handle the LLM side of things, unless you want to try and optimize the amount of tokens which are code vs comments vs explanations and such. (Though you could also just start a new context window with only your code or such)

Re: Fine-Tuning Llama-2: A Comprehensive Case Study for Tailoring Custom Models

#49

Post by purplecats »

jawerty wrote: post #3

Just to add to this, I run through a lot of these topics around fine-tuning Llama 2 on your own dataset (for me it's my own code :P) in a coding live stream a couple weeks ago. All on Colab single GPU Fine-tuning Llama stream: https://www.youtube.com/watch?v=TYgtG2Th6fI&t=2282s I have a couple more one where I do a QLoRa fine tuning session and explain the concepts as a personally self taught engineer (software engin…

really need a simple "put your source stuff in this directory, then press this button, then chat with your contents" type app/module/library.

too much implementation detail required make it inaccessible for any non-significant use case. i imagine privateGpt will get there slowly

Post reply on HN