Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune Google's Gemma 3

unsloth.ai

Post reply on HN
Print view View original

51–60 of 80 posts

Re: Fine-tune Google's Gemma 3

#51

Post by danielhanchen »

moffkalast wrote: post #41
vineyardmike wrote: post #33

Earlier quoted context omitted.

I’ve actually found the opposite. At work, we went from a fine-tuned model to a RAG system for internal and external documentation and a generic coding-focused model for code. Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a cer…

Well, why not both? If you've already got a tuned model why not use RAG on that to get even better results? It already knows the big picture, it just needs the details so it doesn't have to hallucinate them.

Yes RAG combined is pretty cool! Fyi I'm planning to add optimized RAG directly into unsloth as well!

Re: Fine-tune Google's Gemma 3

#52

Post by danielhanchen »

Tostino wrote: post #40
t1amat wrote: post #31

Earlier quoted context omitted.

I would like to see more knowledgeable people with experience talk about this. Is it just a matter of assembling Q/A pairs like: “What’s class X?”, “class X { … }” Do you really need to do this training on the base model instead, which means you have to fine tune chat on it afterward? How does this work?

I've not done fine tuning on code bases but I have done other fine tuning. You will generally get better results when you fine-tune the base model on your data. Since you still want to use it with the chat template in the end, you fine-tune the base model with the chat template with your specific data. From there you'll have a lora that knows your data alright, but still doesn't really work for chatting. You take tha…

Yes!! The trick is the merging of models weights!!

Re: Fine-tune Google's Gemma 3

#53

Post by danielhanchen »

fine_tune wrote: post #34
smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

> I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. I've done it, 1/2 the team though it was great 20% of the time, 1/2 the team hated it from day 0. I used roughly 500K lines of code. > How much effort is required to turn code into something one can use for fine-tuning? Very little to moderate, less than 200 lines of python, QWEM FIM, HF, L…

Oh fill in the middle is definitely smart especially for codebases!!

Re: Fine-tune Google's Gemma 3

#54

Post by danielhanchen »

simonw wrote: post #4
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

I've had trouble getting a great answer to this question - I ask it in various places every month or so, most recently here: https://nitter.net/simonw/status/1895301139819860202 On paper fine tuning smaller models can greatly reduce the cost for a specific task, but I've not heard many real-world success stories around that. I think vision LLMs are one of the most interesting applications here - things like fine-tuni…

Oh there's a lot! Some cool examples I see:

1. Codebases, docs, large corpses of internal datasets - fill in the middle, auto completion etc.

2. I know a tonne of financial institutions use fine-tuning for trading, real time data parsing headline analysis, signal creation etc

3. Distillation is also relatively common - taking outputs of a large model and distilling it to a small model

4. Accuracy increasing is the most important - not cost or latency - we find if you solve the finetuning life cycle ie continuous auto fine-tuning, data filtering, reinforcement learning via DPO, that works well!

5. Lots of organizations use DPO and preference fine-tuning to align models since they have tonnes of feedback data!

6. Yep vision fine-tuning! For eg medical diagnosis, docs, qa on pics etc

7. And obviously large model labs finetune all base models ie chatgpt4.5 is a finetune of a base model

8. Finally reasoning finetuning via GRPO is very cool! If you have inputs and outputs but no labelled cot in between, GRPO is the way to go! Custom reward functions by companies!

Re: Fine-tune Google's Gemma 3

#55

Post by danielhanchen »

bryan0 wrote: post #13

Are people fine-tuning LLMs on their local machines with a single GPU? What are people using to scale their training to multiple nodes / gpus? I've been playing around with Hugging Face Estimators in sagemaker.huggingface but not sure if there are better options for this?

For experimentation and smaller models, single gpu is the way to go! Tbh I normally find most people to spend the majority of their time on datasets, training loss convergence issues etc!

But if its helpful I was thinking about spinning up a platform for something like that!

Re: Fine-tune Google's Gemma 3

#56

Post by danielhanchen »

samspenc wrote: post #17
bryan0 wrote: post #13

Are people fine-tuning LLMs on their local machines with a single GPU? What are people using to scale their training to multiple nodes / gpus? I've been playing around with Hugging Face Estimators in sagemaker.huggingface but not sure if there are better options for this?

It takes a significant amount of time (few hours) on a single consumer GPU, even 4090 / 5090, on personal machines. I think most people use online services like runpod, vast ai, etc to rent out high-powered H100 and similar GPUs for a few cents per hour, run the fine-tuning / training there, and just use local GPUs for inference on those fine-tuned models generated on cloud-rented instances.

It used to be that way! Interestingly I find people in large orgs and the general enthusiast don't mind waiting - memory usage and quality are more important factors!

Re: Fine-tune Google's Gemma 3

#57

Post by simonw »

danielhanchen wrote: post #54
simonw wrote: post #4

Earlier quoted context omitted.

I've had trouble getting a great answer to this question - I ask it in various places every month or so, most recently here: https://nitter.net/simonw/status/1895301139819860202 On paper fine tuning smaller models can greatly reduce the cost for a specific task, but I've not heard many real-world success stories around that. I think vision LLMs are one of the most interesting applications here - things like fine-tuni…

Oh there's a lot! Some cool examples I see: 1. Codebases, docs, large corpses of internal datasets - fill in the middle, auto completion etc. 2. I know a tonne of financial institutions use fine-tuning for trading, real time data parsing headline analysis, signal creation etc 3. Distillation is also relatively common - taking outputs of a large model and distilling it to a small model 4. Accuracy increasing is the mo…

"Codebases, docs, large corpses of internal datasets"

I still haven't seen a convincing demo of using fine-tuning to "teach" a model new information from additional documents. I'd love to see one.

(Closest I've come to that is I heard a rumor that Jane Street have fine-tuned an LLM for OCaml)

Re: Fine-tune Google's Gemma 3

#58

Post by wahnfrieden »

vineyardmike wrote: post #33
smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

I’ve actually found the opposite. At work, we went from a fine-tuned model to a RAG system for internal and external documentation and a generic coding-focused model for code. Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a cer…

Is anyone having success with iteratively feeding chunks of code (or other documents) to LLM for search? I understand 'haystack' issues with LLMs are quite bad, but RAG is quite bad too and a lot of that haystack research seems to be with feeding very large contexts in.

Re: Fine-tune Google's Gemma 3

#59

Post by adefa »

simonw wrote: post #57
danielhanchen wrote: post #54

Earlier quoted context omitted.

Oh there's a lot! Some cool examples I see: 1. Codebases, docs, large corpses of internal datasets - fill in the middle, auto completion etc. 2. I know a tonne of financial institutions use fine-tuning for trading, real time data parsing headline analysis, signal creation etc 3. Distillation is also relatively common - taking outputs of a large model and distilling it to a small model 4. Accuracy increasing is the mo…

"Codebases, docs, large corpses of internal datasets" I still haven't seen a convincing demo of using fine-tuning to "teach" a model new information from additional documents. I'd love to see one. (Closest I've come to that is I heard a rumor that Jane Street have fine-tuned an LLM for OCaml)

Here is a small LLM I trained to output dollars and cents from a verbal numeric amount:

https://huggingface.co/TrevorJS/check-amount-deverbalizer-sm...

Re: Fine-tune Google's Gemma 3

#60

Post by yroc92 »

smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

We are at Scribe[1]. We do it to make sense of knowledge workflows on computers to predict the next step in the process (our software points out where in the DOM a user might need to interact with next). We fine tune with tons of JSON data and DOM data. I’m sure doing it with code no more complicated.

[1] https://scribehow.com/library/scribe-agent

Post reply on HN