Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 270M: Compact model for hyper-efficient AI

developers.googleblog.com

Post reply on HN
Print view View original

161–170 of 325 posts

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#161

Post by mrbonner »

canyon289 wrote: post #89
simonw wrote: post #34

Earlier quoted context omitted.

Do you have any practical examples of fine-tuned variants of this that you can share? A description would be great, but a demo or even downloadable model weights (GGUF ideally) would be even better.

We obviously need to create a pelican bicycle svg finetune ;) If you want to try this out I'd be thrilled to do it with you, I genuinely am curious how well this model can perform if specialized on that task. A couple colleagues of mine posted an example of finetuning a model to take on persona's for videogame NPCs. They have experience working with folks in the game industry and a use case like this is suitable for…

Do you know that hardware required to fine-tune this model? I'm asking on behave of us GPU starve folks

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#162

Post by garbageman »

andrewstuart wrote: post #113
dist-epoch wrote: post #88

Earlier quoted context omitted.

Here you go, one click installer - https://lmstudio.ai

I’m talking about the supplier doing the packaging.

Then use ChatGPT/Gemini/Claude on your phone.

They are giving it away for free - if you NEED a local LLM, the least you can do is spend the 2 minutes to download LMSTudio and pick a model.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#164

Post by canyon289 »

miohtama wrote: post #91

Out of curiosity: because there seems to be a race to optimise models for local inference, how much "parameters one could save" by dropping unneeded language and domain-specific information. Like, can you have a model that is English-only, but does more with the same amount of parameters if Chinese and European languages are dropped from the training?

This is a key question we faced when building this model. It comes down to basically to "how good" to you need to be at "how many things". We had to make some choices with this model and do our best to maximize performance in those areas.

To answer this more precisely its a matter of choosing different data and training regimes and checking performance with evals.

And to make this fully concrete you're welcome to give it a try! Train this model on a taskset of your choice and measure the performance tradeoffs. You'll get a good sense of how LLM capabilities shift

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#165

Post by canyon289 »

perching_aix wrote: post #127
dmayle wrote: post #114

Earlier quoted context omitted.

Did you finetune it before trying? Docs here: https://ai.google.dev/gemma/docs/core/huggingface_text_full_...

Thanks, will check that out as well tomorrow or during the weekend!

If you know you want JSON for sure constrained decoding in an inference framework will help. The model is just one part of an overall inference system. I hope this model paired with other tools help you get done whatever it is you're looking to get done

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#166

Post by ondra »

teraflop wrote: post #116
cristyansv wrote: post #71

Earlier quoted context omitted.

But in your prompts you're trying to assess knowledge, and this model isn't suited for that use case as mentioned in the blog post: > "it can execute tasks like text classification and data extraction with remarkable accuracy, speed, and cost-effectiveness."

Yeah, but if it has in its context window: > List in order the tallest mountains on earth from 1 to 5 > 1. Mount Everest 2. Mount K2 3. Mount Sahel 4. Mount Fuji 5. Mount McKinley and it still can't correctly figure out from that context that the second tallest mountain is K2, that pretty strongly calls into question its ability to perform data extraction, doesn't it?

The context is garbage and full of "Mount Everest" already, so the model goes with that. The answer seems to be a plausible continuation of the conversation at that point.

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#168

Post by marcyb5st »

whinvik wrote: post #23

Curious. Are there real world usecases where people have finetuned such tiny models and put them into production.

I built a reranker for a RAG system using a tiny model. After the candidate generation (i.e. vector search + BM25) and business logic filters/ACL checks the remainder of the chunks went through a model that given the user query told you whether or not the chunk was really relevant. That hit production, but once the context size of models grew that particular piece was discarded as passing everything yielded better results and prices (the fact that prices of input tokens went down also played a role I am sure).

So only for a while, but it still counts :)

Re: Gemma 3 270M: Compact model for hyper-efficient AI

#170

Post by canyon289 »

mrbonner wrote: post #161
canyon289 wrote: post #89

Earlier quoted context omitted.

We obviously need to create a pelican bicycle svg finetune ;) If you want to try this out I'd be thrilled to do it with you, I genuinely am curious how well this model can perform if specialized on that task. A couple colleagues of mine posted an example of finetuning a model to take on persona's for videogame NPCs. They have experience working with folks in the game industry and a use case like this is suitable for…

Do you know that hardware required to fine-tune this model? I'm asking on behave of us GPU starve folks

A free colab. Here's a link, you can finetune the model in ~5 minutes in this example, and I encourage you to try your own

https://ai.google.dev/gemma/docs/core/huggingface_text_full_...

Post reply on HN