Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tune Google's Gemma 3

unsloth.ai

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Fine-tune Google's Gemma 3

#21

Post by m101 »

317070 wrote: post #18
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

I've been finetuning these models since before chatGPT, and the one lesson I've learned is that by the time you have set up everything to fine-tune a model, you can expect a newer model to do as well with prompt-tuning. So, unless you hope to stay at the fore front (e.g. to be ahead of competitors), there has been no real reason to finetune for the last 4 years, at best you could hope to stay about 1-3 months ahead,…

I feel like this is true but would be great if you could provide examples so we could get a better idea of why you think/know this.

Re: Fine-tune Google's Gemma 3

#22

Post by Diederich »

netdur wrote: post #19
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

I have documents from the last 50 years that I need to digitalize, millions of them written in old Arabic. The OCR is not accurate due to handwritten documents, so I need to fine-tune a model on around 300k pairs of texts (OCR output and manually corrected versions)

This sounds very interesting; can you share more? Thanks!

Re: Fine-tune Google's Gemma 3

#23

Post by tough »

refulgentis wrote: post #10
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

IMHO the biggest factor holding that back is how rushed and distanced these model releases are, still. Both Phi-4-mini and Gemma 3 were released recently. Phi-4's damn close to a good, real, model release. Microsoft's done a great job of iterating. Gemma 3's an excellent, intelligent, model, but it's got a gaping blind spot: tool-calling / JSON output. There was a vague quick handwave about it in some PR, a PM/eng on…

could one train now a gemma 3 fine tune for tool use?

found this on HF https://huggingface.co/ZySec-AI/gemma-3-27b-tools

Re: Fine-tune Google's Gemma 3

#24

Post by _ea1k »

bryan0 wrote: post #13

Are people fine-tuning LLMs on their local machines with a single GPU? What are people using to scale their training to multiple nodes / gpus? I've been playing around with Hugging Face Estimators in sagemaker.huggingface but not sure if there are better options for this?

For experimentation? Absolutely. It can often be done overnight for smaller models and reasonably sized GPUs (24GB+).

It'd become a lot less practical with huge datasets, but I'd guess that a lot of fine tuning tasks aren't really that large.

Re: Fine-tune Google's Gemma 3

#25

Post by curious_cat_163 »

simonw wrote: post #15
naveen99 wrote: post #12

Earlier quoted context omitted.

If you have the resources to fine tune, you have the resources to run inference on fine tuned model. If you want to scale up and down on demand, you can just fine tune on openai and google cloud as well.

> If you have the resources to fine tune, you have the resources to run inference on fine tuned model. I don't think that's true. I can fine tune a model by renting a few A100s for a few hours, total cost in the double digit dollars. It's a one-time cost. Running inference with the resulting model for a production application could cost single digit dollars per hour, which adds up to hundreds or even thousands of dol…

This assumes that inference is needed 24/7.

That may or may not be true for use-cases that require asynchronous, bulk inference _and_ require some task-specific post-training.

FWIW, my approach towards tasks like the above is to

1. start with using an off-the-shelf LM API until

2. one figures out (using evals that capture product intent) what the failure modes are (there always are some) and then

3. post-train against those (using the evals)

Re: Fine-tune Google's Gemma 3

#26

Post by anon373839 »

minimaxir wrote: post #8
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

Finetuning is easy and worthwhile, especially with LoRAs as these Unsloth demos do. The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable. In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.

There are inference providers such as Together AI that will serve your LoRA adapters at no extra cost above the model price. Then, there’s basically no difference between using your fine-tuned model or an API model off the shelf (except for the benefits you get from fine-tuning).

Re: Fine-tune Google's Gemma 3

#27

Post by smokel »

I'm interested to know if anyone is using fine-tuning to train a model on proprietary or in-house codebases and documentation.

RAG solutions seem to have their limitations, and fine-tuning might be a more effective approach.

How much effort is required to turn code into something one can use for fine-tuning?

Re: Fine-tune Google's Gemma 3

#28

Post by qudat »

minimaxir wrote: post #8
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

Finetuning is easy and worthwhile, especially with LoRAs as these Unsloth demos do. The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable. In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.

For self-hosting I've been using https://tuns.sh which is a tunneling solution using SSH. It works great for prototyping and I've been using it to host open-webui

Re: Fine-tune Google's Gemma 3

#29

Post by ekojs »

minimaxir wrote: post #8
rockwotj wrote: post #2

is anyone outside of the research labs fine tuning models for production use cases? I have been seeing more people just using foundational models off the shelf especially in light of a new advancement that seems to come every few months

Finetuning is easy and worthwhile, especially with LoRAs as these Unsloth demos do. The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable. In practice prompt engineering and few-shot prompting with modern LLMs, due to their strong-and-only-getting-better-over-time prompt adherence, tends to be more pragmatic.

> The bottleneck then becomes how to self-host the finetuned model in a way that's cost-effective and scalable

It's not actually that expensive and hard. For narrow usecases, you can produce 4-bit quantized fine-tunes that perform as well as the full model. Hosting the 4-bit quantized version can be done on relatively low cost. You can use A40 or RTX 3090 on Runpod for ~$300/month.

Re: Fine-tune Google's Gemma 3

#30

Post by arkmm »

nwienert wrote: post #20
317070 wrote: post #18

Earlier quoted context omitted.

I've been finetuning these models since before chatGPT, and the one lesson I've learned is that by the time you have set up everything to fine-tune a model, you can expect a newer model to do as well with prompt-tuning. So, unless you hope to stay at the fore front (e.g. to be ahead of competitors), there has been no real reason to finetune for the last 4 years, at best you could hope to stay about 1-3 months ahead,…

It feels like there should be a service where I just drag drop a folder of examples and it fine tunes the latest DeepSeek or whatever for me and even can host it for me at some cost. I'd pay for that immediately, but last I checked there was nothing that really did that well (would love to be wrong).

There are some options out there, depending on what type of task you're trying to fine tune. I think RL finetuning for DeepSeek e.g. isn't well developed yet, but you can finetune a small LLama model (~3B params) for classification or extraction tasks and it works really well. What sort of tasks were you looking at finetuning for?
Post reply on HN