Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune Google's Gemma 3

unsloth.ai

Post reply on HN
Print view View original

41–50 of 80 posts

Re: Fine-tune Google's Gemma 3

#41

Post by moffkalast »

vineyardmike wrote: post #33
smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

I’ve actually found the opposite. At work, we went from a fine-tuned model to a RAG system for internal and external documentation and a generic coding-focused model for code. Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a cer…

Well, why not both? If you've already got a tuned model why not use RAG on that to get even better results? It already knows the big picture, it just needs the details so it doesn't have to hallucinate them.

Re: Fine-tune Google's Gemma 3

#42

Post by msp26 »

deet wrote: post #35
simonw wrote: post #4

Earlier quoted context omitted.

I've had trouble getting a great answer to this question - I ask it in various places every month or so, most recently here: https://nitter.net/simonw/status/1895301139819860202 On paper fine tuning smaller models can greatly reduce the cost for a specific task, but I've not heard many real-world success stories around that. I think vision LLMs are one of the most interesting applications here - things like fine-tuni…

Vision LLMs are definitely an interesting application. At Avy.ai we're running small (2B-7B, quantized) vision models as part of a Mac desktop application for understanding what someone is working on in the moment, to offer them related information and actions. We found that the raw results in understanding the images with a light LORA fine tune are not substantially different -- but the ease of getting a small model…

Was constrained decoding not enough to force the output to be in a specific format?

Re: Fine-tune Google's Gemma 3

#43

Post by deet »

msp26 wrote: post #42
deet wrote: post #35

Earlier quoted context omitted.

Vision LLMs are definitely an interesting application. At Avy.ai we're running small (2B-7B, quantized) vision models as part of a Mac desktop application for understanding what someone is working on in the moment, to offer them related information and actions. We found that the raw results in understanding the images with a light LORA fine tune are not substantially different -- but the ease of getting a small model…

Was constrained decoding not enough to force the output to be in a specific format?

Using a grammar to force decoding say valid JSON would work, but that hasn't always been available in the implementations we've been using (like MLX). Solvable by software engineering and adding that to the decoders in those frameworks, but fine tuning has been effective without that work.

The bigger thing though was getting the models to have the appropriate levels of verbosity and detail in their ouput which fine tuning made more consistent.

Re: Fine-tune Google's Gemma 3

#44

Post by huqedato »

Great article, but I didn't see anything about the costs.

I'm particularly interested in this aspect because we're considering fine-tuning Gemma 3, but our budget is tight. We're looking into (real-world) cost estimates for this approach.

Re: Fine-tune Google's Gemma 3

#45

Post by flakiness »

huqedato wrote: post #44

Great article, but I didn't see anything about the costs. I'm particularly interested in this aspect because we're considering fine-tuning Gemma 3, but our budget is tight. We're looking into (real-world) cost estimates for this approach.

> This also means Colab Notebooks with free Tesla T4 GPUs also work!

My understanding is that they don't charge these by themselves although you might have to pay Colab fee to Google.

They charge higher end models it seems https://unsloth.ai/pricing

Re: Fine-tune Google's Gemma 3

#46

Post by danielhanchen »

huqedato wrote: post #44

Great article, but I didn't see anything about the costs. I'm particularly interested in this aspect because we're considering fine-tuning Gemma 3, but our budget is tight. We're looking into (real-world) cost estimates for this approach.

Oh hey! For now we don't have a platform, so we generally tell folks to use Colab free gpus! Kaggle also has 30 hours for free per week! I put links for kaggle here: https://docs.unsloth.ai/get-started/unsloth-notebooks

Re: Fine-tune Google's Gemma 3

#47

Post by danielhanchen »

flakiness wrote: post #45
huqedato wrote: post #44

Great article, but I didn't see anything about the costs. I'm particularly interested in this aspect because we're considering fine-tuning Gemma 3, but our budget is tight. We're looking into (real-world) cost estimates for this approach.

> This also means Colab Notebooks with free Tesla T4 GPUs also work! My understanding is that they don't charge these by themselves although you might have to pay Colab fee to Google. They charge higher end models it seems https://unsloth.ai/pricing

Yep via Colab for now! If its popular I can spin up a deployment fine-tuning system!

Re: Fine-tune Google's Gemma 3

#48

Post by danielhanchen »

t1amat wrote: post #31
smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

I would like to see more knowledgeable people with experience talk about this. Is it just a matter of assembling Q/A pairs like: “What’s class X?”, “class X { … }” Do you really need to do this training on the base model instead, which means you have to fine tune chat on it afterward? How does this work?

Yes qa pairs does work - I found training your dataset concatted with general datasets to work well!

Re: Fine-tune Google's Gemma 3

#49

Post by danielhanchen »

smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

We see a lot of this in large orgs! The main issue imo is actually the selection of chat templates - there's a lot of people who use a template for finetuning then totally forget to use it for finetuning.

A lot of financial, legal and health companies do fine-tuning! Reasoning finetuning via GRPO is also very powerful since you don't need any cot data in between! Just inputs and outputs!

Re: Fine-tune Google's Gemma 3

#50

Post by danielhanchen »

vineyardmike wrote: post #33
smokel wrote: post #27

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation. RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach. How much effort is required to turn code into something one can use for fine-tuning?

I’ve actually found the opposite. At work, we went from a fine-tuned model to a RAG system for internal and external documentation and a generic coding-focused model for code. Fine tuning against in-house code seems like a small gain over a base model and search. It’s unlikely your code is unique and special and big enough that it’s hard to get results from a base model. You’ll be pinned to a certain version of a cer…

RAG definitely is helpful! Fine-tuning imo is extremely powerful but it's still relatively alchemy - technically gpt4, Claude any large model is a finetune of a base model! Reasoning finetuning is also very powerful!

Tbh the hardest part is the lifecycle - ie new data, updating, serving etc - that seems to be the biggest issue

Post reply on HN